2023-06-22T17:04:51.375 INFO:root:teuthology version: 0.0.1.dev128+g076bbeb 2023-06-22T17:04:51.376 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311823 --name yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi --verbose --owner scheduled_yuriw@teuthology --description fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/pjd} wsync/{no}} -- /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311823/orig.config.yaml 2023-06-22T17:04:51.432 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-22T17:04:51.529 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311823 branch: wip-yuri4-testing-2023-06-21-0846-quincy description: fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/pjd} wsync/{no}} email: yweinste@redhat.com first_in_suite: false job_id: '7311823' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.4' overrides: admin_socket: branch: wip-yuri4-testing-2023-06-21-0846-quincy ceph: cephfs: session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 fuse default permissions: false fuse set user groups: true rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 install: ceph: flavor: default sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - nowsync selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2023-06-21-0846-quincy sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 9037 sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c sleep_before_teardown: 0 subset: 111/120000 suite: fs suite_branch: wip-yuri4-testing-2023-06-21-0846-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c targets: smithi039.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZePTAHvZbx6OtHBjisBZNYiX7R2eUrvpKqn+Br+kxucLXXbrPLIiyoul/s4+IiB6NnHtVu7M+igECv+GnMtvYW5y5TzivzjheGR2LeVgFfDJqqPvsmEnwNJoWfWLOukfpwG6aZPUmo0jBvsEci3DqgG1DmkwCTvh+oRxHeWsKvMaTASKYgM1a6ww25p7y3mjHPetaI1ca7YNLPpkiZc4XHPNfkyUO0suFsQbfK9lVqIbUq0yKnMfB0jAr1l5tQkwUX6+KMaFNS+tW2jwt+vh93hxWjvvjeUB9F2A+fhAZbRoeDd4pujduoXtC6OfWFvLQJfkzb1wQfyFHam1BoOSRaTw5E2MsOSv0PCYJnZ55PRt94caNrDQwVKQ+3r1eonhOcAFJmh58wpL8EolFQ9WGHSxMY8Xeb3Y0fLjykOa+W3QSdqZQC+1E0GGEj4+NKmRvMrKLzGyyoyehGFiEHIjkiYgEe0gsPk+C4XO3aPxeC/R0lz9fMUgo/CtpRixlu8M= smithi101.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCUPiLlj4GcVhnjueUMi4vjBDm/pnWv0LBKz/KVj6tO1gukUoK2H12nzaWpbkwGVTSHKsjiAjeCt6kOtOOiyLZ+ZsdDQ6C7puyz55nSLjSQ7Am/ZVR6W988kRDKTzNhT0SbxcfEQcp7Q3eiA4v4keIBKdVJKnJDP3z/jEVH0Q8oAhSvOebfa6ArD2yc78QRbcnjHp+97O6qYUhzBM236l+gTP+x9E4GCk9t0pP7y/5bFxiSr2eYkNyFSfDJQGkeVsobpoTTMKze8QANo0vwxGLqYNjlS/EDNSLYW/mEzX2nSO0tXYRPWwW54mH7H7m3GACQ2xC6+AfD8vtKmIQNvXagzMqSPIZa4moVRLEkmoWJWibHpOw+DPEEVR5LwzICUaxM2PCRq4YVmVKFP8/FyaJMr7RvWrDIOZvF93oZBEE5cJmAC9s5Bb4KetwGiKHORyiZ1YQ6ip3QtzZOHYNeVqbAz54HHaNN4dLHjO69AbLFaBTKVKOgM1FLaBVoPFqKE08= smithi133.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkDVaRpyyMnXCswECTHN+Peaetgd+6nvXoDYKD6SLV4BaVriJBgQ4t/LC6XWMd08u74t3H63suGOlTv6BNj8S/PP1eXvqiX6XOkgH5N+ziVwNs9Ce1infLiT8/wwauHNuOBUbJYwt4BY2HmrAviuVhyRy+sH4PkvxzOAn/6YakuxoL5w8EMNX66lUfeiglR3mUKs4X1A5jNGkxfhkYfcAjXjcujgViZdDc6HhtN42INflCGfA+ukylfyo1/o8n52fEe31ClsNyrKw/TinChNWmSX8aReQKA2PiAOwk7moUOxKNQKD6lQwblKNVDs6HiOG9TOoL91azLey2f3Alsi45i2BCKOpF9YBQv1hxX/YtPCGcRyzB2SsdpSNeS8uaHXdOolCE5RB0ZKCX59mUipQYyGmiHtgI0KdV6CJJffZOwDtdGFgfYuYQxJEK9M6dJ6BezyFXz0QrUpd1M8vMiNGI4f0TYORr6wqHqCLiw0AHLUINTncfewceZDD3vbllQAU= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - ceph-fuse: null - check-counter: {} - workunit: clients: all: - suites/pjd.sh timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 076bbebc42a14f7d568aaa78eabb0038327bcb23 timestamp: 2023-06-22_15:25:25 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-06-22T17:04:51.530 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa; will attempt to use it 2023-06-22T17:04:51.530 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks 2023-06-22T17:04:51.531 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-06-22T17:04:51.532 INFO:teuthology.task.internal:Checking packages... 2023-06-22T17:04:51.565 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '42ddc46e6e6582d8e29a5a982a81886fe028be2c' 2023-06-22T17:04:51.566 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-06-22T17:04:51.567 INFO:teuthology.packaging:ref: None 2023-06-22T17:04:51.567 INFO:teuthology.packaging:tag: None 2023-06-22T17:04:51.567 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-06-21-0846-quincy 2023-06-22T17:04:51.567 INFO:teuthology.packaging:sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-22T17:04:51.568 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2023-06-21-0846-quincy 2023-06-22T17:04:51.781 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-719.g42ddc46e 2023-06-22T17:04:51.782 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-06-22T17:04:51.789 INFO:teuthology.task.internal:no buildpackages task found 2023-06-22T17:04:51.789 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-06-22T17:04:51.829 INFO:teuthology.task.internal:Saving configuration 2023-06-22T17:04:51.847 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-06-22T17:04:51.853 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-22T17:04:51.880 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi039.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311823', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-22 16:58:52.438243', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOWjB7zS7XAhI0Prlv+dW3UHGuDTbBHxLOuMJ4fM24aG5+qUBVY4DRbGSe2LPmDw3/R7uLnavAWBOWvFDynz+h0='} 2023-06-22T17:04:51.903 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi101.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311823', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-22 16:58:52.436353', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHDMAM/SQjLwi3eX+SOLyl2kHN3Bj//jwUMuyROZwwXAFb9y5Cbhr463gKCgZG7+AYBoglKY+St8tRxq0o9BHFk='} 2023-06-22T17:04:51.928 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi133.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311823', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-22 16:58:52.439724', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD/Pb5uIagwTrNe6Ght94pPT6ldMuRFI4DVrVVCilr87c9BZxYxRh//G4w/+znp0Z0KQE0xNpT5URVh1YifzpEE='} 2023-06-22T17:04:51.929 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-06-22T17:04:51.935 INFO:teuthology.task.internal:roles: ubuntu@smithi039.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2023-06-22T17:04:51.935 INFO:teuthology.task.internal:roles: ubuntu@smithi101.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2023-06-22T17:04:51.935 INFO:teuthology.task.internal:roles: ubuntu@smithi133.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2023-06-22T17:04:51.936 INFO:teuthology.run_tasks:Running task console_log... 2023-06-22T17:04:52.080 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe536e250d0>, signals=[15]) 2023-06-22T17:04:52.081 INFO:teuthology.run_tasks:Running task internal.connect... 2023-06-22T17:04:52.089 INFO:teuthology.task.internal:Opening connections... 2023-06-22T17:04:52.090 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi039.front.sepia.ceph.com 2023-06-22T17:04:52.092 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T17:04:52.163 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi101.front.sepia.ceph.com 2023-06-22T17:04:52.163 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi101.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T17:04:52.231 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi133.front.sepia.ceph.com 2023-06-22T17:04:52.232 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T17:04:52.301 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-06-22T17:04:52.307 DEBUG:teuthology.orchestra.run.smithi039:> uname -m 2023-06-22T17:04:52.325 INFO:teuthology.orchestra.run.smithi039.stdout:x86_64 2023-06-22T17:04:52.325 DEBUG:teuthology.orchestra.run.smithi039:> cat /etc/os-release 2023-06-22T17:04:52.382 INFO:teuthology.orchestra.run.smithi039.stdout:NAME="Red Hat Enterprise Linux" 2023-06-22T17:04:52.382 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION="8.4 (Ootpa)" 2023-06-22T17:04:52.382 INFO:teuthology.orchestra.run.smithi039.stdout:ID="rhel" 2023-06-22T17:04:52.382 INFO:teuthology.orchestra.run.smithi039.stdout:ID_LIKE="fedora" 2023-06-22T17:04:52.383 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION_ID="8.4" 2023-06-22T17:04:52.383 INFO:teuthology.orchestra.run.smithi039.stdout:PLATFORM_ID="platform:el8" 2023-06-22T17:04:52.383 INFO:teuthology.orchestra.run.smithi039.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-06-22T17:04:52.383 INFO:teuthology.orchestra.run.smithi039.stdout:ANSI_COLOR="0;31" 2023-06-22T17:04:52.383 INFO:teuthology.orchestra.run.smithi039.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-06-22T17:04:52.383 INFO:teuthology.orchestra.run.smithi039.stdout:HOME_URL="https://www.redhat.com/" 2023-06-22T17:04:52.383 INFO:teuthology.orchestra.run.smithi039.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-06-22T17:04:52.383 INFO:teuthology.orchestra.run.smithi039.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-06-22T17:04:52.384 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:04:52.384 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-06-22T17:04:52.384 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-06-22T17:04:52.384 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-06-22T17:04:52.384 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-06-22T17:04:52.385 INFO:teuthology.lock.ops:Updating smithi039.front.sepia.ceph.com on lock server 2023-06-22T17:04:52.408 DEBUG:teuthology.orchestra.run.smithi101:> uname -m 2023-06-22T17:04:52.426 INFO:teuthology.orchestra.run.smithi101.stdout:x86_64 2023-06-22T17:04:52.426 DEBUG:teuthology.orchestra.run.smithi101:> cat /etc/os-release 2023-06-22T17:04:52.483 INFO:teuthology.orchestra.run.smithi101.stdout:NAME="Red Hat Enterprise Linux" 2023-06-22T17:04:52.483 INFO:teuthology.orchestra.run.smithi101.stdout:VERSION="8.4 (Ootpa)" 2023-06-22T17:04:52.484 INFO:teuthology.orchestra.run.smithi101.stdout:ID="rhel" 2023-06-22T17:04:52.484 INFO:teuthology.orchestra.run.smithi101.stdout:ID_LIKE="fedora" 2023-06-22T17:04:52.484 INFO:teuthology.orchestra.run.smithi101.stdout:VERSION_ID="8.4" 2023-06-22T17:04:52.484 INFO:teuthology.orchestra.run.smithi101.stdout:PLATFORM_ID="platform:el8" 2023-06-22T17:04:52.484 INFO:teuthology.orchestra.run.smithi101.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-06-22T17:04:52.484 INFO:teuthology.orchestra.run.smithi101.stdout:ANSI_COLOR="0;31" 2023-06-22T17:04:52.484 INFO:teuthology.orchestra.run.smithi101.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-06-22T17:04:52.485 INFO:teuthology.orchestra.run.smithi101.stdout:HOME_URL="https://www.redhat.com/" 2023-06-22T17:04:52.485 INFO:teuthology.orchestra.run.smithi101.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-06-22T17:04:52.485 INFO:teuthology.orchestra.run.smithi101.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-06-22T17:04:52.485 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-06-22T17:04:52.485 INFO:teuthology.orchestra.run.smithi101.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-06-22T17:04:52.485 INFO:teuthology.orchestra.run.smithi101.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-06-22T17:04:52.485 INFO:teuthology.orchestra.run.smithi101.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-06-22T17:04:52.486 INFO:teuthology.orchestra.run.smithi101.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-06-22T17:04:52.486 INFO:teuthology.lock.ops:Updating smithi101.front.sepia.ceph.com on lock server 2023-06-22T17:04:52.508 DEBUG:teuthology.orchestra.run.smithi133:> uname -m 2023-06-22T17:04:52.525 INFO:teuthology.orchestra.run.smithi133.stdout:x86_64 2023-06-22T17:04:52.526 DEBUG:teuthology.orchestra.run.smithi133:> cat /etc/os-release 2023-06-22T17:04:52.581 INFO:teuthology.orchestra.run.smithi133.stdout:NAME="Red Hat Enterprise Linux" 2023-06-22T17:04:52.582 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION="8.4 (Ootpa)" 2023-06-22T17:04:52.582 INFO:teuthology.orchestra.run.smithi133.stdout:ID="rhel" 2023-06-22T17:04:52.583 INFO:teuthology.orchestra.run.smithi133.stdout:ID_LIKE="fedora" 2023-06-22T17:04:52.583 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION_ID="8.4" 2023-06-22T17:04:52.583 INFO:teuthology.orchestra.run.smithi133.stdout:PLATFORM_ID="platform:el8" 2023-06-22T17:04:52.583 INFO:teuthology.orchestra.run.smithi133.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-06-22T17:04:52.583 INFO:teuthology.orchestra.run.smithi133.stdout:ANSI_COLOR="0;31" 2023-06-22T17:04:52.583 INFO:teuthology.orchestra.run.smithi133.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-06-22T17:04:52.583 INFO:teuthology.orchestra.run.smithi133.stdout:HOME_URL="https://www.redhat.com/" 2023-06-22T17:04:52.584 INFO:teuthology.orchestra.run.smithi133.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-06-22T17:04:52.584 INFO:teuthology.orchestra.run.smithi133.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-06-22T17:04:52.584 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-06-22T17:04:52.584 INFO:teuthology.orchestra.run.smithi133.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-06-22T17:04:52.584 INFO:teuthology.orchestra.run.smithi133.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-06-22T17:04:52.584 INFO:teuthology.orchestra.run.smithi133.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-06-22T17:04:52.585 INFO:teuthology.orchestra.run.smithi133.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-06-22T17:04:52.585 INFO:teuthology.lock.ops:Updating smithi133.front.sepia.ceph.com on lock server 2023-06-22T17:04:52.609 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-06-22T17:04:52.629 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-06-22T17:04:52.636 INFO:teuthology.task.internal:Checking for old test directory... 2023-06-22T17:04:52.636 DEBUG:teuthology.orchestra.run.smithi039:> test '!' -e /home/ubuntu/cephtest 2023-06-22T17:04:52.639 DEBUG:teuthology.orchestra.run.smithi101:> test '!' -e /home/ubuntu/cephtest 2023-06-22T17:04:52.641 DEBUG:teuthology.orchestra.run.smithi133:> test '!' -e /home/ubuntu/cephtest 2023-06-22T17:04:52.657 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-06-22T17:04:52.663 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-06-22T17:04:52.664 DEBUG:teuthology.orchestra.run.smithi039:> test -z $(ls -A /var/lib/ceph) 2023-06-22T17:04:52.695 DEBUG:teuthology.orchestra.run.smithi101:> test -z $(ls -A /var/lib/ceph) 2023-06-22T17:04:52.697 DEBUG:teuthology.orchestra.run.smithi133:> test -z $(ls -A /var/lib/ceph) 2023-06-22T17:04:52.710 INFO:teuthology.orchestra.run.smithi039.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-06-22T17:04:52.711 INFO:teuthology.orchestra.run.smithi101.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-06-22T17:04:52.713 INFO:teuthology.orchestra.run.smithi133.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-06-22T17:04:52.714 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-06-22T17:04:52.800 INFO:teuthology.run_tasks:Running task kernel... 2023-06-22T17:04:52.816 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-06-22T17:04:52.817 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-06-22T17:04:52.817 DEBUG:teuthology.orchestra.run.smithi039:> test -f /run/.containerenv -o -f /.dockerenv 2023-06-22T17:04:52.818 DEBUG:teuthology.orchestra.run.smithi101:> test -f /run/.containerenv -o -f /.dockerenv 2023-06-22T17:04:52.818 DEBUG:teuthology.orchestra.run.smithi133:> test -f /run/.containerenv -o -f /.dockerenv 2023-06-22T17:04:52.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-22T17:04:52.834 DEBUG:teuthology.orchestra.run.smithi133:> uname -r 2023-06-22T17:04:52.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-22T17:04:52.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-22T17:04:52.836 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2023-06-22T17:04:52.837 DEBUG:teuthology.orchestra.run.smithi101:> uname -r 2023-06-22T17:04:52.890 INFO:teuthology.orchestra.run.smithi133.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-06-22T17:04:52.892 INFO:teuthology.task.kernel:Running kernel on smithi133: 4.18.0-305.25.1.el8_4.x86_64 2023-06-22T17:04:52.892 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum install -y kernel 2023-06-22T17:04:52.894 INFO:teuthology.orchestra.run.smithi039.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-06-22T17:04:52.894 INFO:teuthology.orchestra.run.smithi101.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-06-22T17:04:52.894 INFO:teuthology.task.kernel:Running kernel on smithi039: 4.18.0-305.25.1.el8_4.x86_64 2023-06-22T17:04:52.894 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum install -y kernel 2023-06-22T17:04:52.895 INFO:teuthology.task.kernel:Running kernel on smithi101: 4.18.0-305.25.1.el8_4.x86_64 2023-06-22T17:04:52.896 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum install -y kernel 2023-06-22T17:04:54.264 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-06-22T17:04:54.265 INFO:teuthology.orchestra.run.smithi039.stdout:Unable to read consumer identity 2023-06-22T17:04:54.290 INFO:teuthology.orchestra.run.smithi133.stdout:Updating Subscription Management repositories. 2023-06-22T17:04:54.291 INFO:teuthology.orchestra.run.smithi133.stdout:Unable to read consumer identity 2023-06-22T17:04:54.475 INFO:teuthology.orchestra.run.smithi101.stdout:Updating Subscription Management repositories. 2023-06-22T17:04:54.476 INFO:teuthology.orchestra.run.smithi101.stdout:Unable to read consumer identity 2023-06-22T17:04:57.978 INFO:teuthology.orchestra.run.smithi101.stdout:Copr repo for python3-asyncssh owned by ceph 1.6 kB/s | 3.5 kB 00:02 2023-06-22T17:04:58.003 INFO:teuthology.orchestra.run.smithi133.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2023-06-22T17:04:58.012 INFO:teuthology.orchestra.run.smithi039.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2023-06-22T17:04:59.532 INFO:teuthology.orchestra.run.smithi039.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2023-06-22T17:04:59.533 INFO:teuthology.orchestra.run.smithi133.stdout:Extra Packages for Enterprise Linux 10 MB/s | 16 MB 00:01 2023-06-22T17:05:00.786 INFO:teuthology.orchestra.run.smithi101.stdout:Extra Packages for Enterprise Linux 5.6 MB/s | 16 MB 00:02 2023-06-22T17:05:02.715 INFO:teuthology.orchestra.run.smithi133.stdout:lab-extras 4.4 MB/s | 24 kB 00:00 2023-06-22T17:05:02.904 INFO:teuthology.orchestra.run.smithi039.stdout:lab-extras 4.1 MB/s | 24 kB 00:00 2023-06-22T17:05:03.494 INFO:teuthology.orchestra.run.smithi133.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-06-22T17:05:03.494 INFO:teuthology.orchestra.run.smithi133.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-06-22T17:05:03.494 INFO:teuthology.orchestra.run.smithi133.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-06-22T17:05:03.648 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-06-22T17:05:03.649 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2023-06-22T17:05:03.649 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-06-22T17:05:03.764 DEBUG:teuthology.orchestra.run.smithi133:> echo no | sudo yum reinstall kernel || true 2023-06-22T17:05:03.797 INFO:teuthology.orchestra.run.smithi039.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-06-22T17:05:03.797 INFO:teuthology.orchestra.run.smithi039.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-06-22T17:05:03.797 INFO:teuthology.orchestra.run.smithi039.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-06-22T17:05:03.956 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-06-22T17:05:03.957 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2023-06-22T17:05:03.957 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-06-22T17:05:04.082 DEBUG:teuthology.orchestra.run.smithi039:> echo no | sudo yum reinstall kernel || true 2023-06-22T17:05:04.108 INFO:teuthology.orchestra.run.smithi133.stdout:Updating Subscription Management repositories. 2023-06-22T17:05:04.109 INFO:teuthology.orchestra.run.smithi133.stdout:Unable to read consumer identity 2023-06-22T17:05:04.173 INFO:teuthology.orchestra.run.smithi101.stdout:lab-extras 4.4 MB/s | 24 kB 00:00 2023-06-22T17:05:04.398 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:00:02 ago on Thu 22 Jun 2023 05:05:02 PM UTC. 2023-06-22T17:05:04.447 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-06-22T17:05:04.447 INFO:teuthology.orchestra.run.smithi039.stdout:Unable to read consumer identity 2023-06-22T17:05:04.518 INFO:teuthology.orchestra.run.smithi133.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-06-22T17:05:04.518 INFO:teuthology.orchestra.run.smithi133.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-22T17:05:04.519 INFO:teuthology.orchestra.run.smithi133.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-22T17:05:04.546 INFO:teuthology.orchestra.run.smithi133.stderr:Error: No packages marked for reinstall. 2023-06-22T17:05:04.597 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum reinstall -y kernel || true 2023-06-22T17:05:04.752 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:00:02 ago on Thu 22 Jun 2023 05:05:02 PM UTC. 2023-06-22T17:05:04.912 INFO:teuthology.orchestra.run.smithi039.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-06-22T17:05:04.912 INFO:teuthology.orchestra.run.smithi039.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-22T17:05:04.913 INFO:teuthology.orchestra.run.smithi039.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-22T17:05:04.920 INFO:teuthology.orchestra.run.smithi101.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-06-22T17:05:04.920 INFO:teuthology.orchestra.run.smithi101.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-06-22T17:05:04.920 INFO:teuthology.orchestra.run.smithi101.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-06-22T17:05:04.939 INFO:teuthology.orchestra.run.smithi133.stdout:Updating Subscription Management repositories. 2023-06-22T17:05:04.939 INFO:teuthology.orchestra.run.smithi133.stdout:Unable to read consumer identity 2023-06-22T17:05:04.942 INFO:teuthology.orchestra.run.smithi039.stderr:Error: No packages marked for reinstall. 2023-06-22T17:05:04.992 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum reinstall -y kernel || true 2023-06-22T17:05:05.092 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-06-22T17:05:05.093 INFO:teuthology.orchestra.run.smithi101.stdout:Nothing to do. 2023-06-22T17:05:05.093 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-06-22T17:05:05.198 DEBUG:teuthology.orchestra.run.smithi101:> echo no | sudo yum reinstall kernel || true 2023-06-22T17:05:05.227 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:00:03 ago on Thu 22 Jun 2023 05:05:02 PM UTC. 2023-06-22T17:05:05.357 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-06-22T17:05:05.358 INFO:teuthology.orchestra.run.smithi039.stdout:Unable to read consumer identity 2023-06-22T17:05:05.383 INFO:teuthology.orchestra.run.smithi133.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-06-22T17:05:05.384 INFO:teuthology.orchestra.run.smithi133.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-22T17:05:05.384 INFO:teuthology.orchestra.run.smithi133.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-22T17:05:05.398 INFO:teuthology.orchestra.run.smithi133.stderr:Error: No packages marked for reinstall. 2023-06-22T17:05:05.454 DEBUG:teuthology.orchestra.run.smithi133:> rpm -q kernel | sort -rV | head -n 1 2023-06-22T17:05:05.543 INFO:teuthology.orchestra.run.smithi101.stdout:Updating Subscription Management repositories. 2023-06-22T17:05:05.544 INFO:teuthology.orchestra.run.smithi101.stdout:Unable to read consumer identity 2023-06-22T17:05:05.578 INFO:teuthology.orchestra.run.smithi133.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-06-22T17:05:05.578 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-06-22T17:05:05.579 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-06-22T17:05:05.579 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-06-22T17:05:05.579 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-06-22T17:05:05.579 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2023-06-22T17:05:05.579 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi133.front.sepia.ceph.com, path=None, version=distro) 2023-06-22T17:05:05.579 DEBUG:teuthology.orchestra.run.smithi133:> rpm -q kernel | sort -rV | head -n 1 2023-06-22T17:05:05.655 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:00:03 ago on Thu 22 Jun 2023 05:05:02 PM UTC. 2023-06-22T17:05:05.704 INFO:teuthology.orchestra.run.smithi133.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-06-22T17:05:05.704 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-06-22T17:05:05.704 DEBUG:teuthology.orchestra.run.smithi133:> sudo rpm -qi grub2-tools 2023-06-22T17:05:05.732 INFO:teuthology.orchestra.run.smithi133.stdout:Name : grub2-tools 2023-06-22T17:05:05.732 INFO:teuthology.orchestra.run.smithi133.stdout:Epoch : 1 2023-06-22T17:05:05.732 INFO:teuthology.orchestra.run.smithi133.stdout:Version : 2.02 2023-06-22T17:05:05.733 INFO:teuthology.orchestra.run.smithi133.stdout:Release : 99.el8_4.1 2023-06-22T17:05:05.733 INFO:teuthology.orchestra.run.smithi133.stdout:Architecture: x86_64 2023-06-22T17:05:05.733 INFO:teuthology.orchestra.run.smithi133.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-06-22T17:05:05.733 INFO:teuthology.orchestra.run.smithi133.stdout:Group : System Environment/Base 2023-06-22T17:05:05.734 INFO:teuthology.orchestra.run.smithi133.stdout:Size : 9264631 2023-06-22T17:05:05.734 INFO:teuthology.orchestra.run.smithi133.stdout:License : GPLv3+ 2023-06-22T17:05:05.734 INFO:teuthology.orchestra.run.smithi133.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-06-22T17:05:05.735 INFO:teuthology.orchestra.run.smithi133.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-06-22T17:05:05.735 INFO:teuthology.orchestra.run.smithi133.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-06-22T17:05:05.735 INFO:teuthology.orchestra.run.smithi133.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-06-22T17:05:05.735 INFO:teuthology.orchestra.run.smithi133.stdout:Relocations : (not relocatable) 2023-06-22T17:05:05.736 INFO:teuthology.orchestra.run.smithi133.stdout:Packager : Red Hat, Inc. 2023-06-22T17:05:05.736 INFO:teuthology.orchestra.run.smithi133.stdout:Vendor : Red Hat, Inc. 2023-06-22T17:05:05.736 INFO:teuthology.orchestra.run.smithi133.stdout:URL : http://www.gnu.org/software/grub/ 2023-06-22T17:05:05.736 INFO:teuthology.orchestra.run.smithi133.stdout:Summary : Support tools for GRUB. 2023-06-22T17:05:05.737 INFO:teuthology.orchestra.run.smithi133.stdout:Description : 2023-06-22T17:05:05.737 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-06-22T17:05:05.737 INFO:teuthology.orchestra.run.smithi133.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-06-22T17:05:05.738 INFO:teuthology.orchestra.run.smithi133.stdout:customizable bootloader with modular architecture. It supports a rich 2023-06-22T17:05:05.738 INFO:teuthology.orchestra.run.smithi133.stdout:variety of kernel formats, file systems, computer architectures and 2023-06-22T17:05:05.738 INFO:teuthology.orchestra.run.smithi133.stdout:hardware devices. 2023-06-22T17:05:05.738 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-06-22T17:05:05.738 INFO:teuthology.orchestra.run.smithi133.stdout:This subpackage provides tools for support of all platforms. 2023-06-22T17:05:05.739 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-06-22T17:05:05.739 INFO:teuthology.task.kernel:Updating grub on smithi133 to boot 4.18.0-425.3.1.el8.x86_64 2023-06-22T17:05:05.739 DEBUG:teuthology.orchestra.run.smithi133:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-06-22T17:05:05.795 INFO:teuthology.orchestra.run.smithi039.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-06-22T17:05:05.795 INFO:teuthology.orchestra.run.smithi039.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-22T17:05:05.796 INFO:teuthology.orchestra.run.smithi039.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-22T17:05:05.809 INFO:teuthology.orchestra.run.smithi039.stderr:Error: No packages marked for reinstall. 2023-06-22T17:05:05.839 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:00:01 ago on Thu 22 Jun 2023 05:05:04 PM UTC. 2023-06-22T17:05:05.865 DEBUG:teuthology.orchestra.run.smithi039:> rpm -q kernel | sort -rV | head -n 1 2023-06-22T17:05:05.973 INFO:teuthology.orchestra.run.smithi101.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-06-22T17:05:05.973 INFO:teuthology.orchestra.run.smithi101.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-22T17:05:05.973 INFO:teuthology.orchestra.run.smithi101.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-22T17:05:05.995 INFO:teuthology.orchestra.run.smithi039.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-06-22T17:05:05.996 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-06-22T17:05:05.996 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-06-22T17:05:05.996 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-06-22T17:05:05.996 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-06-22T17:05:05.997 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-06-22T17:05:05.997 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi039.front.sepia.ceph.com, path=None, version=distro) 2023-06-22T17:05:05.997 DEBUG:teuthology.orchestra.run.smithi039:> rpm -q kernel | sort -rV | head -n 1 2023-06-22T17:05:06.006 INFO:teuthology.orchestra.run.smithi101.stderr:Error: No packages marked for reinstall. 2023-06-22T17:05:06.055 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum reinstall -y kernel || true 2023-06-22T17:05:06.127 INFO:teuthology.orchestra.run.smithi039.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-06-22T17:05:06.128 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-06-22T17:05:06.128 DEBUG:teuthology.orchestra.run.smithi039:> sudo rpm -qi grub2-tools 2023-06-22T17:05:06.155 INFO:teuthology.orchestra.run.smithi039.stdout:Name : grub2-tools 2023-06-22T17:05:06.156 INFO:teuthology.orchestra.run.smithi039.stdout:Epoch : 1 2023-06-22T17:05:06.156 INFO:teuthology.orchestra.run.smithi039.stdout:Version : 2.02 2023-06-22T17:05:06.156 INFO:teuthology.orchestra.run.smithi039.stdout:Release : 99.el8_4.1 2023-06-22T17:05:06.156 INFO:teuthology.orchestra.run.smithi039.stdout:Architecture: x86_64 2023-06-22T17:05:06.156 INFO:teuthology.orchestra.run.smithi039.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-06-22T17:05:06.156 INFO:teuthology.orchestra.run.smithi039.stdout:Group : System Environment/Base 2023-06-22T17:05:06.156 INFO:teuthology.orchestra.run.smithi039.stdout:Size : 9264631 2023-06-22T17:05:06.157 INFO:teuthology.orchestra.run.smithi039.stdout:License : GPLv3+ 2023-06-22T17:05:06.157 INFO:teuthology.orchestra.run.smithi039.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-06-22T17:05:06.157 INFO:teuthology.orchestra.run.smithi039.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-06-22T17:05:06.157 INFO:teuthology.orchestra.run.smithi039.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-06-22T17:05:06.157 INFO:teuthology.orchestra.run.smithi039.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-06-22T17:05:06.157 INFO:teuthology.orchestra.run.smithi039.stdout:Relocations : (not relocatable) 2023-06-22T17:05:06.157 INFO:teuthology.orchestra.run.smithi039.stdout:Packager : Red Hat, Inc. 2023-06-22T17:05:06.157 INFO:teuthology.orchestra.run.smithi039.stdout:Vendor : Red Hat, Inc. 2023-06-22T17:05:06.158 INFO:teuthology.orchestra.run.smithi039.stdout:URL : http://www.gnu.org/software/grub/ 2023-06-22T17:05:06.158 INFO:teuthology.orchestra.run.smithi039.stdout:Summary : Support tools for GRUB. 2023-06-22T17:05:06.158 INFO:teuthology.orchestra.run.smithi039.stdout:Description : 2023-06-22T17:05:06.158 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:05:06.158 INFO:teuthology.orchestra.run.smithi039.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-06-22T17:05:06.158 INFO:teuthology.orchestra.run.smithi039.stdout:customizable bootloader with modular architecture. It supports a rich 2023-06-22T17:05:06.158 INFO:teuthology.orchestra.run.smithi039.stdout:variety of kernel formats, file systems, computer architectures and 2023-06-22T17:05:06.158 INFO:teuthology.orchestra.run.smithi039.stdout:hardware devices. 2023-06-22T17:05:06.159 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:05:06.159 INFO:teuthology.orchestra.run.smithi039.stdout:This subpackage provides tools for support of all platforms. 2023-06-22T17:05:06.159 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-06-22T17:05:06.160 INFO:teuthology.task.kernel:Updating grub on smithi039 to boot 4.18.0-425.3.1.el8.x86_64 2023-06-22T17:05:06.160 DEBUG:teuthology.orchestra.run.smithi039:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-06-22T17:05:06.397 INFO:teuthology.orchestra.run.smithi101.stdout:Updating Subscription Management repositories. 2023-06-22T17:05:06.397 INFO:teuthology.orchestra.run.smithi101.stdout:Unable to read consumer identity 2023-06-22T17:05:06.677 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:00:02 ago on Thu 22 Jun 2023 05:05:04 PM UTC. 2023-06-22T17:05:06.805 INFO:teuthology.orchestra.run.smithi101.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-06-22T17:05:06.805 INFO:teuthology.orchestra.run.smithi101.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-22T17:05:06.805 INFO:teuthology.orchestra.run.smithi101.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-22T17:05:06.820 INFO:teuthology.orchestra.run.smithi101.stderr:Error: No packages marked for reinstall. 2023-06-22T17:05:06.875 DEBUG:teuthology.orchestra.run.smithi101:> rpm -q kernel | sort -rV | head -n 1 2023-06-22T17:05:06.998 INFO:teuthology.orchestra.run.smithi101.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-06-22T17:05:06.998 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-06-22T17:05:06.998 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-06-22T17:05:06.998 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-06-22T17:05:06.999 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-06-22T17:05:06.999 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-06-22T17:05:06.999 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi101.front.sepia.ceph.com, path=None, version=distro) 2023-06-22T17:05:06.999 DEBUG:teuthology.orchestra.run.smithi101:> rpm -q kernel | sort -rV | head -n 1 2023-06-22T17:05:07.097 INFO:teuthology.orchestra.run.smithi133.stderr:Generating grub configuration file ... 2023-06-22T17:05:07.114 INFO:teuthology.orchestra.run.smithi039.stderr:Generating grub configuration file ... 2023-06-22T17:05:07.123 INFO:teuthology.orchestra.run.smithi101.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-06-22T17:05:07.123 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-06-22T17:05:07.124 DEBUG:teuthology.orchestra.run.smithi101:> sudo rpm -qi grub2-tools 2023-06-22T17:05:07.150 INFO:teuthology.orchestra.run.smithi101.stdout:Name : grub2-tools 2023-06-22T17:05:07.150 INFO:teuthology.orchestra.run.smithi101.stdout:Epoch : 1 2023-06-22T17:05:07.150 INFO:teuthology.orchestra.run.smithi101.stdout:Version : 2.02 2023-06-22T17:05:07.151 INFO:teuthology.orchestra.run.smithi101.stdout:Release : 99.el8_4.1 2023-06-22T17:05:07.151 INFO:teuthology.orchestra.run.smithi101.stdout:Architecture: x86_64 2023-06-22T17:05:07.151 INFO:teuthology.orchestra.run.smithi101.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-06-22T17:05:07.151 INFO:teuthology.orchestra.run.smithi101.stdout:Group : System Environment/Base 2023-06-22T17:05:07.151 INFO:teuthology.orchestra.run.smithi101.stdout:Size : 9264631 2023-06-22T17:05:07.151 INFO:teuthology.orchestra.run.smithi101.stdout:License : GPLv3+ 2023-06-22T17:05:07.151 INFO:teuthology.orchestra.run.smithi101.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-06-22T17:05:07.152 INFO:teuthology.orchestra.run.smithi101.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-06-22T17:05:07.152 INFO:teuthology.orchestra.run.smithi101.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-06-22T17:05:07.152 INFO:teuthology.orchestra.run.smithi101.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-06-22T17:05:07.152 INFO:teuthology.orchestra.run.smithi101.stdout:Relocations : (not relocatable) 2023-06-22T17:05:07.152 INFO:teuthology.orchestra.run.smithi101.stdout:Packager : Red Hat, Inc. 2023-06-22T17:05:07.152 INFO:teuthology.orchestra.run.smithi101.stdout:Vendor : Red Hat, Inc. 2023-06-22T17:05:07.152 INFO:teuthology.orchestra.run.smithi101.stdout:URL : http://www.gnu.org/software/grub/ 2023-06-22T17:05:07.152 INFO:teuthology.orchestra.run.smithi101.stdout:Summary : Support tools for GRUB. 2023-06-22T17:05:07.153 INFO:teuthology.orchestra.run.smithi101.stdout:Description : 2023-06-22T17:05:07.153 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-06-22T17:05:07.153 INFO:teuthology.orchestra.run.smithi101.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-06-22T17:05:07.153 INFO:teuthology.orchestra.run.smithi101.stdout:customizable bootloader with modular architecture. It supports a rich 2023-06-22T17:05:07.153 INFO:teuthology.orchestra.run.smithi101.stdout:variety of kernel formats, file systems, computer architectures and 2023-06-22T17:05:07.153 INFO:teuthology.orchestra.run.smithi101.stdout:hardware devices. 2023-06-22T17:05:07.153 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-06-22T17:05:07.153 INFO:teuthology.orchestra.run.smithi101.stdout:This subpackage provides tools for support of all platforms. 2023-06-22T17:05:07.154 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-06-22T17:05:07.154 INFO:teuthology.task.kernel:Updating grub on smithi101 to boot 4.18.0-425.3.1.el8.x86_64 2023-06-22T17:05:07.155 DEBUG:teuthology.orchestra.run.smithi101:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-06-22T17:05:08.371 INFO:teuthology.orchestra.run.smithi101.stderr:Generating grub configuration file ... 2023-06-22T17:05:09.005 INFO:teuthology.orchestra.run.smithi133.stderr:done 2023-06-22T17:05:09.007 DEBUG:teuthology.orchestra.run.smithi133:> mktemp 2023-06-22T17:05:09.039 INFO:teuthology.orchestra.run.smithi133.stdout:/tmp/tmp.esbsAzkkub 2023-06-22T17:05:09.039 DEBUG:teuthology.orchestra.run.smithi133:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.esbsAzkkub 2023-06-22T17:05:09.108 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 0666 /tmp/tmp.esbsAzkkub 2023-06-22T17:05:09.148 INFO:teuthology.orchestra.run.smithi039.stderr:done 2023-06-22T17:05:09.149 DEBUG:teuthology.orchestra.run.smithi039:> mktemp 2023-06-22T17:05:09.164 INFO:teuthology.orchestra.run.smithi039.stdout:/tmp/tmp.33Z1b0xR0l 2023-06-22T17:05:09.164 DEBUG:teuthology.orchestra.run.smithi039:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.33Z1b0xR0l 2023-06-22T17:05:09.243 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0666 /tmp/tmp.33Z1b0xR0l 2023-06-22T17:05:09.254 DEBUG:teuthology.orchestra.remote:smithi133:/tmp/tmp.esbsAzkkub is 6KB 2023-06-22T17:05:09.302 DEBUG:teuthology.orchestra.run.smithi133:> rm -fr /tmp/tmp.esbsAzkkub 2023-06-22T17:05:09.315 DEBUG:teuthology.orchestra.run.smithi133:> sudo /bin/ls /boot/loader/entries || true 2023-06-22T17:05:09.381 INFO:teuthology.orchestra.run.smithi133.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-06-22T17:05:09.382 INFO:teuthology.orchestra.run.smithi133.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-06-22T17:05:09.382 INFO:teuthology.orchestra.run.smithi133.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-06-22T17:05:09.382 INFO:teuthology.orchestra.run.smithi133.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-06-22T17:05:09.383 DEBUG:teuthology.orchestra.run.smithi133:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-06-22T17:05:09.400 DEBUG:teuthology.orchestra.remote:smithi039:/tmp/tmp.33Z1b0xR0l is 6KB 2023-06-22T17:05:09.419 DEBUG:teuthology.orchestra.run.smithi039:> rm -fr /tmp/tmp.33Z1b0xR0l 2023-06-22T17:05:09.433 DEBUG:teuthology.orchestra.run.smithi039:> sudo /bin/ls /boot/loader/entries || true 2023-06-22T17:05:09.503 INFO:teuthology.orchestra.run.smithi039.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-06-22T17:05:09.503 INFO:teuthology.orchestra.run.smithi039.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-06-22T17:05:09.503 INFO:teuthology.orchestra.run.smithi039.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-06-22T17:05:09.503 INFO:teuthology.orchestra.run.smithi039.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-06-22T17:05:09.504 DEBUG:teuthology.orchestra.run.smithi039:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-06-22T17:05:09.573 DEBUG:teuthology.orchestra.run.smithi133:> sudo shutdown -r now 2023-06-22T17:05:09.660 DEBUG:teuthology.orchestra.run.smithi039:> sudo shutdown -r now 2023-06-22T17:05:10.482 INFO:teuthology.orchestra.run.smithi101.stderr:done 2023-06-22T17:05:10.483 DEBUG:teuthology.orchestra.run.smithi101:> mktemp 2023-06-22T17:05:10.496 INFO:teuthology.orchestra.run.smithi101.stdout:/tmp/tmp.2L4PDS91jI 2023-06-22T17:05:10.497 DEBUG:teuthology.orchestra.run.smithi101:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.2L4PDS91jI 2023-06-22T17:05:10.566 DEBUG:teuthology.orchestra.run.smithi101:> sudo chmod 0666 /tmp/tmp.2L4PDS91jI 2023-06-22T17:05:10.741 DEBUG:teuthology.orchestra.remote:smithi101:/tmp/tmp.2L4PDS91jI is 6KB 2023-06-22T17:05:10.791 DEBUG:teuthology.orchestra.run.smithi101:> rm -fr /tmp/tmp.2L4PDS91jI 2023-06-22T17:05:10.805 DEBUG:teuthology.orchestra.run.smithi101:> sudo /bin/ls /boot/loader/entries || true 2023-06-22T17:05:10.880 INFO:teuthology.orchestra.run.smithi101.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-06-22T17:05:10.880 INFO:teuthology.orchestra.run.smithi101.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-06-22T17:05:10.881 INFO:teuthology.orchestra.run.smithi101.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-06-22T17:05:10.881 INFO:teuthology.orchestra.run.smithi101.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-06-22T17:05:10.882 DEBUG:teuthology.orchestra.run.smithi101:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-06-22T17:05:11.066 DEBUG:teuthology.orchestra.run.smithi101:> sudo shutdown -r now 2023-06-22T17:05:39.604 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2023-06-22T17:05:39.604 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2023-06-22T17:05:39.605 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T17:05:39.663 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-06-22T17:05:39.663 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2023-06-22T17:05:39.664 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T17:05:41.068 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-06-22T17:05:41.069 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi101.front.sepia.ceph.com' 2023-06-22T17:05:41.069 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi101.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T17:06:14.256 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2023-06-22T17:06:39.628 DEBUG:teuthology.orchestra.remote:timed out 2023-06-22T17:06:41.072 DEBUG:teuthology.orchestra.remote:timed out 2023-06-22T17:06:44.260 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2023-06-22T17:06:44.261 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T17:07:09.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2023-06-22T17:07:09.649 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T17:07:10.028 DEBUG:teuthology.orchestra.run.smithi133:> true 2023-06-22T17:07:10.277 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi133.front.sepia.ceph.com' 2023-06-22T17:07:10.278 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "4.18.0-425.3.1.el8.x86_64"... 2023-06-22T17:07:10.278 DEBUG:teuthology.orchestra.run.smithi133:> uname -r 2023-06-22T17:07:10.332 INFO:teuthology.orchestra.run.smithi133.stdout:4.18.0-425.3.1.el8.x86_64 2023-06-22T17:07:10.332 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-06-22T17:07:10.333 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-06-22T17:07:10.333 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-06-22T17:07:11.073 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi101.front.sepia.ceph.com' 2023-06-22T17:07:11.074 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi101.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T17:07:11.334 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2023-06-22T17:07:11.334 DEBUG:teuthology.orchestra.run.smithi133:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-06-22T17:07:11.419 INFO:teuthology.orchestra.run.smithi133.stdout:ttyS1 2023-06-22T17:07:11.433 DEBUG:teuthology.parallel:result is None 2023-06-22T17:07:11.443 DEBUG:teuthology.orchestra.run.smithi101:> true 2023-06-22T17:07:11.718 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi101.front.sepia.ceph.com' 2023-06-22T17:07:11.718 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-425.3.1.el8.x86_64"... 2023-06-22T17:07:11.719 DEBUG:teuthology.orchestra.run.smithi101:> uname -r 2023-06-22T17:07:11.733 INFO:teuthology.orchestra.run.smithi101.stdout:4.18.0-425.3.1.el8.x86_64 2023-06-22T17:07:11.734 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-06-22T17:07:11.734 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-06-22T17:07:11.734 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-06-22T17:07:12.735 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-06-22T17:07:12.735 DEBUG:teuthology.orchestra.run.smithi101:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-06-22T17:07:12.807 INFO:teuthology.orchestra.run.smithi101.stdout:ttyS1 2023-06-22T17:07:12.820 DEBUG:teuthology.parallel:result is None 2023-06-22T17:07:44.262 DEBUG:teuthology.orchestra.remote:timed out 2023-06-22T17:08:14.264 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2023-06-22T17:08:14.265 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T17:08:14.557 DEBUG:teuthology.orchestra.run.smithi039:> true 2023-06-22T17:08:14.837 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi039.front.sepia.ceph.com' 2023-06-22T17:08:14.838 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-425.3.1.el8.x86_64"... 2023-06-22T17:08:14.838 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2023-06-22T17:08:14.854 INFO:teuthology.orchestra.run.smithi039.stdout:4.18.0-425.3.1.el8.x86_64 2023-06-22T17:08:14.855 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-06-22T17:08:14.855 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-06-22T17:08:14.855 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-06-22T17:08:15.856 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-06-22T17:08:15.857 DEBUG:teuthology.orchestra.run.smithi039:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-06-22T17:08:15.946 INFO:teuthology.orchestra.run.smithi039.stdout:ttyS1 2023-06-22T17:08:15.959 DEBUG:teuthology.parallel:result is None 2023-06-22T17:08:15.959 INFO:teuthology.run_tasks:Running task internal.base... 2023-06-22T17:08:15.966 INFO:teuthology.task.internal:Creating test directory... 2023-06-22T17:08:15.967 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-06-22T17:08:15.969 DEBUG:teuthology.orchestra.run.smithi101:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-06-22T17:08:15.971 DEBUG:teuthology.orchestra.run.smithi133:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-06-22T17:08:15.989 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-06-22T17:08:16.004 INFO:teuthology.run_tasks:Running task internal.archive... 2023-06-22T17:08:16.019 INFO:teuthology.task.internal:Creating archive directory... 2023-06-22T17:08:16.020 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-06-22T17:08:16.025 DEBUG:teuthology.orchestra.run.smithi101:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-06-22T17:08:16.029 DEBUG:teuthology.orchestra.run.smithi133:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-06-22T17:08:16.070 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-06-22T17:08:16.077 INFO:teuthology.task.internal:Enabling coredump saving... 2023-06-22T17:08:16.077 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-06-22T17:08:16.088 DEBUG:teuthology.orchestra.run.smithi101:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-06-22T17:08:16.096 DEBUG:teuthology.orchestra.run.smithi133:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-06-22T17:08:16.115 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-22T17:08:16.132 INFO:teuthology.orchestra.run.smithi101.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-22T17:08:16.132 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-22T17:08:16.133 INFO:teuthology.orchestra.run.smithi133.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-22T17:08:16.139 INFO:teuthology.orchestra.run.smithi101.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-22T17:08:16.144 INFO:teuthology.orchestra.run.smithi133.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-22T17:08:16.145 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-06-22T17:08:16.152 INFO:teuthology.task.internal:Configuring sudo... 2023-06-22T17:08:16.153 DEBUG:teuthology.orchestra.run.smithi039:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-06-22T17:08:16.175 DEBUG:teuthology.orchestra.run.smithi101:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-06-22T17:08:16.183 DEBUG:teuthology.orchestra.run.smithi133:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-06-22T17:08:16.212 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-06-22T17:08:16.221 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-06-22T17:08:16.222 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-06-22T17:08:16.242 DEBUG:teuthology.orchestra.run.smithi101:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-06-22T17:08:16.250 DEBUG:teuthology.orchestra.run.smithi133:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-06-22T17:08:16.267 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-22T17:08:16.343 DEBUG:teuthology.orchestra.run.smithi039:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-22T17:08:16.382 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-22T17:08:16.468 DEBUG:teuthology.orchestra.run.smithi039:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-22T17:08:16.504 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-22T17:08:16.505 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-06-22T17:08:16.569 DEBUG:teuthology.orchestra.run.smithi101:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-22T17:08:16.615 DEBUG:teuthology.orchestra.run.smithi101:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-22T17:08:16.654 DEBUG:teuthology.orchestra.run.smithi101:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-22T17:08:16.737 DEBUG:teuthology.orchestra.run.smithi101:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-22T17:08:16.778 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-06-22T17:08:16.778 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-06-22T17:08:16.842 DEBUG:teuthology.orchestra.run.smithi133:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-22T17:08:16.888 DEBUG:teuthology.orchestra.run.smithi133:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-22T17:08:16.932 DEBUG:teuthology.orchestra.run.smithi133:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-22T17:08:17.018 DEBUG:teuthology.orchestra.run.smithi133:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-22T17:08:17.054 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-06-22T17:08:17.054 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-06-22T17:08:17.118 DEBUG:teuthology.orchestra.run.smithi039:> sudo service rsyslog restart 2023-06-22T17:08:17.120 DEBUG:teuthology.orchestra.run.smithi101:> sudo service rsyslog restart 2023-06-22T17:08:17.122 DEBUG:teuthology.orchestra.run.smithi133:> sudo service rsyslog restart 2023-06-22T17:08:17.166 INFO:teuthology.orchestra.run.smithi039.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-22T17:08:17.195 INFO:teuthology.orchestra.run.smithi101.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-22T17:08:17.228 INFO:teuthology.orchestra.run.smithi133.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-22T17:08:17.679 INFO:teuthology.run_tasks:Running task internal.timer... 2023-06-22T17:08:17.687 INFO:teuthology.task.internal:Starting timer... 2023-06-22T17:08:17.687 INFO:teuthology.run_tasks:Running task pcp... 2023-06-22T17:08:17.704 INFO:teuthology.run_tasks:Running task selinux... 2023-06-22T17:08:17.712 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-06-22T17:08:17.713 DEBUG:teuthology.orchestra.run.smithi039:> sudo service auditd rotate 2023-06-22T17:08:17.770 INFO:teuthology.orchestra.run.smithi039.stdout:Rotating logs: [ OK ] 2023-06-22T17:08:17.772 DEBUG:teuthology.orchestra.run.smithi101:> sudo service auditd rotate 2023-06-22T17:08:17.840 INFO:teuthology.orchestra.run.smithi101.stdout:Rotating logs: [ OK ] 2023-06-22T17:08:17.841 DEBUG:teuthology.orchestra.run.smithi133:> sudo service auditd rotate 2023-06-22T17:08:17.897 INFO:teuthology.orchestra.run.smithi133.stdout:Rotating logs: [ OK ] 2023-06-22T17:08:17.899 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-06-22T17:08:17.900 DEBUG:teuthology.orchestra.run.smithi039:> /usr/sbin/getenforce 2023-06-22T17:08:17.935 INFO:teuthology.orchestra.run.smithi039.stdout:Permissive 2023-06-22T17:08:17.935 DEBUG:teuthology.orchestra.run.smithi101:> /usr/sbin/getenforce 2023-06-22T17:08:17.959 INFO:teuthology.orchestra.run.smithi101.stdout:Permissive 2023-06-22T17:08:17.960 DEBUG:teuthology.orchestra.run.smithi133:> /usr/sbin/getenforce 2023-06-22T17:08:18.057 INFO:teuthology.orchestra.run.smithi133.stdout:Permissive 2023-06-22T17:08:18.057 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi039.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi101.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi133.front.sepia.ceph.com': 'permissive'} 2023-06-22T17:08:18.058 DEBUG:teuthology.orchestra.run.smithi039:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-22T17:08:18.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-22T17:08:18.085 DEBUG:teuthology.orchestra.run.smithi101:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-22T17:08:18.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-22T17:08:18.113 DEBUG:teuthology.orchestra.run.smithi133:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-22T17:08:18.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-22T17:08:18.139 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-06-22T17:08:18.139 DEBUG:teuthology.orchestra.run.smithi039:> sudo /usr/sbin/setenforce permissive 2023-06-22T17:08:18.165 DEBUG:teuthology.orchestra.run.smithi101:> sudo /usr/sbin/setenforce permissive 2023-06-22T17:08:18.197 DEBUG:teuthology.orchestra.run.smithi133:> sudo /usr/sbin/setenforce permissive 2023-06-22T17:08:18.223 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-06-22T17:08:18.233 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-06-22T17:08:18.234 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-06-22T17:08:18.253 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-06-22T17:08:18.255 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi039.front.sepia.ceph.com,smithi101.front.sepia.ceph.com,smithi133.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-06-22T17:16:13.774 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi039.front.sepia.ceph.com'), Remote(name='ubuntu@smithi101.front.sepia.ceph.com'), Remote(name='ubuntu@smithi133.front.sepia.ceph.com')] 2023-06-22T17:16:13.776 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2023-06-22T17:16:13.778 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T17:16:13.845 DEBUG:teuthology.orchestra.run.smithi039:> true 2023-06-22T17:16:13.911 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi101.front.sepia.ceph.com' 2023-06-22T17:16:13.911 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi101.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T17:16:13.979 DEBUG:teuthology.orchestra.run.smithi101:> true 2023-06-22T17:16:14.042 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2023-06-22T17:16:14.043 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T17:16:14.111 DEBUG:teuthology.orchestra.run.smithi133:> true 2023-06-22T17:16:14.179 INFO:teuthology.run_tasks:Running task clock... 2023-06-22T17:16:14.193 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-06-22T17:16:14.194 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-06-22T17:16:14.195 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-22T17:16:14.198 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-06-22T17:16:14.198 DEBUG:teuthology.orchestra.run.smithi101:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-22T17:16:14.201 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-06-22T17:16:14.202 DEBUG:teuthology.orchestra.run.smithi133:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-22T17:16:14.227 INFO:teuthology.orchestra.run.smithi101.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-06-22T17:16:14.227 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-06-22T17:16:14.241 INFO:teuthology.orchestra.run.smithi101.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-06-22T17:16:14.242 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-06-22T17:16:14.246 INFO:teuthology.orchestra.run.smithi133.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-06-22T17:16:14.261 INFO:teuthology.orchestra.run.smithi133.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-06-22T17:16:14.274 INFO:teuthology.orchestra.run.smithi101.stderr:sudo: ntpd: command not found 2023-06-22T17:16:14.278 INFO:teuthology.orchestra.run.smithi039.stderr:sudo: ntpd: command not found 2023-06-22T17:16:14.286 INFO:teuthology.orchestra.run.smithi101.stdout:506 Cannot talk to daemon 2023-06-22T17:16:14.288 INFO:teuthology.orchestra.run.smithi039.stdout:506 Cannot talk to daemon 2023-06-22T17:16:14.294 INFO:teuthology.orchestra.run.smithi133.stderr:sudo: ntpd: command not found 2023-06-22T17:16:14.297 INFO:teuthology.orchestra.run.smithi101.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-06-22T17:16:14.302 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-06-22T17:16:14.304 INFO:teuthology.orchestra.run.smithi133.stdout:506 Cannot talk to daemon 2023-06-22T17:16:14.311 INFO:teuthology.orchestra.run.smithi101.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-06-22T17:16:14.316 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-06-22T17:16:14.317 INFO:teuthology.orchestra.run.smithi133.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-06-22T17:16:14.331 INFO:teuthology.orchestra.run.smithi133.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-06-22T17:16:14.386 INFO:teuthology.orchestra.run.smithi101.stderr:bash: ntpq: command not found 2023-06-22T17:16:14.387 INFO:teuthology.orchestra.run.smithi039.stderr:bash: ntpq: command not found 2023-06-22T17:16:14.388 INFO:teuthology.orchestra.run.smithi101.stdout:210 Number of sources = 4 2023-06-22T17:16:14.388 INFO:teuthology.orchestra.run.smithi101.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-22T17:16:14.388 INFO:teuthology.orchestra.run.smithi101.stdout:=============================================================================== 2023-06-22T17:16:14.389 INFO:teuthology.orchestra.run.smithi101.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T17:16:14.389 INFO:teuthology.orchestra.run.smithi101.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T17:16:14.389 INFO:teuthology.orchestra.run.smithi101.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T17:16:14.389 INFO:teuthology.orchestra.run.smithi101.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T17:16:14.390 INFO:teuthology.orchestra.run.smithi039.stdout:210 Number of sources = 4 2023-06-22T17:16:14.390 INFO:teuthology.orchestra.run.smithi039.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-22T17:16:14.391 INFO:teuthology.orchestra.run.smithi039.stdout:=============================================================================== 2023-06-22T17:16:14.391 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T17:16:14.391 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T17:16:14.391 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T17:16:14.391 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T17:16:14.395 INFO:teuthology.orchestra.run.smithi133.stderr:bash: ntpq: command not found 2023-06-22T17:16:14.398 INFO:teuthology.orchestra.run.smithi133.stdout:210 Number of sources = 4 2023-06-22T17:16:14.399 INFO:teuthology.orchestra.run.smithi133.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-22T17:16:14.399 INFO:teuthology.orchestra.run.smithi133.stdout:=============================================================================== 2023-06-22T17:16:14.399 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T17:16:14.400 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T17:16:14.400 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T17:16:14.400 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T17:16:14.401 INFO:teuthology.run_tasks:Running task pexec... 2023-06-22T17:16:14.410 INFO:teuthology.task.pexec:Executing custom commands... 2023-06-22T17:16:14.411 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi039.front.sepia.ceph.com 2023-06-22T17:16:14.411 DEBUG:teuthology.orchestra.run.smithi039:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-06-22T17:16:14.411 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi101.front.sepia.ceph.com 2023-06-22T17:16:14.412 DEBUG:teuthology.orchestra.run.smithi101:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-06-22T17:16:14.412 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi133.front.sepia.ceph.com 2023-06-22T17:16:14.412 DEBUG:teuthology.orchestra.run.smithi133:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-06-22T17:16:14.823 INFO:teuthology.orchestra.run.smithi101.stdout:Updating Subscription Management repositories. 2023-06-22T17:16:14.829 INFO:teuthology.orchestra.run.smithi133.stdout:Updating Subscription Management repositories. 2023-06-22T17:16:14.843 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-06-22T17:16:15.618 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:35 ago on Thu 22 Jun 2023 05:14:40 PM UTC. 2023-06-22T17:16:15.703 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:01:38 ago on Thu 22 Jun 2023 05:14:37 PM UTC. 2023-06-22T17:16:15.712 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:26 ago on Thu 22 Jun 2023 05:14:49 PM UTC. 2023-06-22T17:16:17.611 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-06-22T17:16:17.612 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-06-22T17:16:17.612 INFO:teuthology.orchestra.run.smithi133.stdout: Package Architecture Version Repository Size 2023-06-22T17:16:17.612 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-06-22T17:16:17.612 INFO:teuthology.orchestra.run.smithi133.stdout:Resetting modules: 2023-06-22T17:16:17.612 INFO:teuthology.orchestra.run.smithi133.stdout: container-tools 2023-06-22T17:16:17.613 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-06-22T17:16:17.613 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-06-22T17:16:17.613 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-06-22T17:16:17.613 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-06-22T17:16:17.690 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-06-22T17:16:17.691 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-06-22T17:16:17.692 INFO:teuthology.orchestra.run.smithi101.stdout: Package Architecture Version Repository Size 2023-06-22T17:16:17.692 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-06-22T17:16:17.692 INFO:teuthology.orchestra.run.smithi101.stdout:Resetting modules: 2023-06-22T17:16:17.693 INFO:teuthology.orchestra.run.smithi101.stdout: container-tools 2023-06-22T17:16:17.693 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-06-22T17:16:17.693 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-06-22T17:16:17.693 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-06-22T17:16:17.694 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-06-22T17:16:17.872 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-06-22T17:16:17.873 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-06-22T17:16:17.873 INFO:teuthology.orchestra.run.smithi039.stdout: Package Architecture Version Repository Size 2023-06-22T17:16:17.873 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-06-22T17:16:17.873 INFO:teuthology.orchestra.run.smithi039.stdout:Resetting modules: 2023-06-22T17:16:17.874 INFO:teuthology.orchestra.run.smithi039.stdout: container-tools 2023-06-22T17:16:17.874 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:16:17.874 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-06-22T17:16:17.874 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-06-22T17:16:17.874 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:16:18.466 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:38 ago on Thu 22 Jun 2023 05:14:40 PM UTC. 2023-06-22T17:16:18.634 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:01:41 ago on Thu 22 Jun 2023 05:14:37 PM UTC. 2023-06-22T17:16:18.864 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:29 ago on Thu 22 Jun 2023 05:14:49 PM UTC. 2023-06-22T17:16:20.277 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-06-22T17:16:20.519 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-06-22T17:16:20.713 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-06-22T17:16:20.721 INFO:teuthology.orchestra.run.smithi133.stdout:Updating Subscription Management repositories. 2023-06-22T17:16:20.948 INFO:teuthology.orchestra.run.smithi101.stdout:Updating Subscription Management repositories. 2023-06-22T17:16:21.175 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-06-22T17:16:21.450 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:41 ago on Thu 22 Jun 2023 05:14:40 PM UTC. 2023-06-22T17:16:21.764 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:01:44 ago on Thu 22 Jun 2023 05:14:37 PM UTC. 2023-06-22T17:16:22.028 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:32 ago on Thu 22 Jun 2023 05:14:49 PM UTC. 2023-06-22T17:16:23.433 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-06-22T17:16:23.481 INFO:teuthology.orchestra.run.smithi133.stdout:========================================================================================================= 2023-06-22T17:16:23.482 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2023-06-22T17:16:23.482 INFO:teuthology.orchestra.run.smithi133.stdout:========================================================================================================= 2023-06-22T17:16:23.482 INFO:teuthology.orchestra.run.smithi133.stdout:Installing group/module packages: 2023-06-22T17:16:23.482 INFO:teuthology.orchestra.run.smithi133.stdout: buildah x86_64 1.21.4-2.module+el8.4.0+12050+ef972f71 rhel-8-for-x86_64-appstream-rpms 8.2 M 2023-06-22T17:16:23.482 INFO:teuthology.orchestra.run.smithi133.stdout: cockpit-podman noarch 32-2.module+el8.4.0+11990+22932769 rhel-8-for-x86_64-appstream-rpms 410 k 2023-06-22T17:16:23.482 INFO:teuthology.orchestra.run.smithi133.stdout: crun x86_64 0.20.1-1.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 192 k 2023-06-22T17:16:23.483 INFO:teuthology.orchestra.run.smithi133.stdout: skopeo x86_64 1:1.3.1-5.module+el8.4.0+11990+22932769 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-06-22T17:16:23.483 INFO:teuthology.orchestra.run.smithi133.stdout: toolbox noarch 0.0.8-1.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 16 k 2023-06-22T17:16:23.483 INFO:teuthology.orchestra.run.smithi133.stdout: udica noarch 0.2.4-2.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 51 k 2023-06-22T17:16:23.483 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2023-06-22T17:16:23.483 INFO:teuthology.orchestra.run.smithi133.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-06-22T17:16:23.483 INFO:teuthology.orchestra.run.smithi133.stdout:Installing module profiles: 2023-06-22T17:16:23.483 INFO:teuthology.orchestra.run.smithi133.stdout: container-tools/common 2023-06-22T17:16:23.483 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-06-22T17:16:23.484 INFO:teuthology.orchestra.run.smithi133.stdout:Enabling module streams: 2023-06-22T17:16:23.484 INFO:teuthology.orchestra.run.smithi133.stdout: container-tools rhel8 2023-06-22T17:16:23.484 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-06-22T17:16:23.484 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-06-22T17:16:23.484 INFO:teuthology.orchestra.run.smithi133.stdout:========================================================================================================= 2023-06-22T17:16:23.484 INFO:teuthology.orchestra.run.smithi133.stdout:Install 7 Packages 2023-06-22T17:16:23.484 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-06-22T17:16:23.485 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 16 M 2023-06-22T17:16:23.485 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 57 M 2023-06-22T17:16:23.485 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-06-22T17:16:23.519 INFO:teuthology.orchestra.run.smithi133.stdout:(1/7): cockpit-podman-32-2.module+el8.4.0+11990 12 MB/s | 410 kB 00:00 2023-06-22T17:16:23.524 INFO:teuthology.orchestra.run.smithi133.stdout:(2/7): yajl-2.1.0-10.el8.x86_64.rpm 1.0 MB/s | 41 kB 00:00 2023-06-22T17:16:23.526 INFO:teuthology.orchestra.run.smithi133.stdout:(3/7): crun-0.20.1-1.module+el8.4.0+11822+6cc1e 4.4 MB/s | 192 kB 00:00 2023-06-22T17:16:23.536 INFO:teuthology.orchestra.run.smithi133.stdout:(4/7): toolbox-0.0.8-1.module+el8.4.0+11822+6cc 903 kB/s | 16 kB 00:00 2023-06-22T17:16:23.554 INFO:teuthology.orchestra.run.smithi133.stdout:(5/7): udica-0.2.4-2.module+el8.4.0+11822+6cc1e 2.9 MB/s | 51 kB 00:00 2023-06-22T17:16:23.629 INFO:teuthology.orchestra.run.smithi133.stdout:(6/7): skopeo-1.3.1-5.module+el8.4.0+11990+2293 67 MB/s | 7.0 MB 00:00 2023-06-22T17:16:23.655 INFO:teuthology.orchestra.run.smithi133.stdout:(7/7): buildah-1.21.4-2.module+el8.4.0+12050+ef 64 MB/s | 8.2 MB 00:00 2023-06-22T17:16:23.655 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-06-22T17:16:23.655 INFO:teuthology.orchestra.run.smithi133.stdout:Total 92 MB/s | 16 MB 00:00 2023-06-22T17:16:23.661 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-06-22T17:16:23.678 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-06-22T17:16:23.678 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-06-22T17:16:23.762 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-06-22T17:16:23.810 INFO:teuthology.orchestra.run.smithi101.stdout:========================================================================================================= 2023-06-22T17:16:23.811 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repository Size 2023-06-22T17:16:23.811 INFO:teuthology.orchestra.run.smithi101.stdout:========================================================================================================= 2023-06-22T17:16:23.811 INFO:teuthology.orchestra.run.smithi101.stdout:Installing group/module packages: 2023-06-22T17:16:23.811 INFO:teuthology.orchestra.run.smithi101.stdout: buildah x86_64 1.21.4-2.module+el8.4.0+12050+ef972f71 rhel-8-for-x86_64-appstream-rpms 8.2 M 2023-06-22T17:16:23.811 INFO:teuthology.orchestra.run.smithi101.stdout: cockpit-podman noarch 32-2.module+el8.4.0+11990+22932769 rhel-8-for-x86_64-appstream-rpms 410 k 2023-06-22T17:16:23.811 INFO:teuthology.orchestra.run.smithi101.stdout: crun x86_64 0.20.1-1.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 192 k 2023-06-22T17:16:23.811 INFO:teuthology.orchestra.run.smithi101.stdout: skopeo x86_64 1:1.3.1-5.module+el8.4.0+11990+22932769 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-06-22T17:16:23.812 INFO:teuthology.orchestra.run.smithi101.stdout: toolbox noarch 0.0.8-1.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 16 k 2023-06-22T17:16:23.812 INFO:teuthology.orchestra.run.smithi101.stdout: udica noarch 0.2.4-2.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 51 k 2023-06-22T17:16:23.812 INFO:teuthology.orchestra.run.smithi101.stdout:Installing dependencies: 2023-06-22T17:16:23.812 INFO:teuthology.orchestra.run.smithi101.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-06-22T17:16:23.812 INFO:teuthology.orchestra.run.smithi101.stdout:Installing module profiles: 2023-06-22T17:16:23.812 INFO:teuthology.orchestra.run.smithi101.stdout: container-tools/common 2023-06-22T17:16:23.812 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-06-22T17:16:23.813 INFO:teuthology.orchestra.run.smithi101.stdout:Enabling module streams: 2023-06-22T17:16:23.813 INFO:teuthology.orchestra.run.smithi101.stdout: container-tools rhel8 2023-06-22T17:16:23.813 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-06-22T17:16:23.813 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-06-22T17:16:23.813 INFO:teuthology.orchestra.run.smithi101.stdout:========================================================================================================= 2023-06-22T17:16:23.813 INFO:teuthology.orchestra.run.smithi101.stdout:Install 7 Packages 2023-06-22T17:16:23.813 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-06-22T17:16:23.814 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 16 M 2023-06-22T17:16:23.814 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 57 M 2023-06-22T17:16:23.814 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-06-22T17:16:23.843 INFO:teuthology.orchestra.run.smithi101.stdout:(1/7): crun-0.20.1-1.module+el8.4.0+11822+6cc1e 6.2 MB/s | 192 kB 00:00 2023-06-22T17:16:23.848 INFO:teuthology.orchestra.run.smithi101.stdout:(2/7): yajl-2.1.0-10.el8.x86_64.rpm 1.1 MB/s | 41 kB 00:00 2023-06-22T17:16:23.856 INFO:teuthology.orchestra.run.smithi101.stdout:(3/7): cockpit-podman-32-2.module+el8.4.0+11990 9.3 MB/s | 410 kB 00:00 2023-06-22T17:16:23.860 INFO:teuthology.orchestra.run.smithi101.stdout:(4/7): toolbox-0.0.8-1.module+el8.4.0+11822+6cc 938 kB/s | 16 kB 00:00 2023-06-22T17:16:23.874 INFO:teuthology.orchestra.run.smithi101.stdout:(5/7): udica-0.2.4-2.module+el8.4.0+11822+6cc1e 3.4 MB/s | 51 kB 00:00 2023-06-22T17:16:23.948 INFO:teuthology.orchestra.run.smithi101.stdout:(6/7): skopeo-1.3.1-5.module+el8.4.0+11990+2293 70 MB/s | 7.0 MB 00:00 2023-06-22T17:16:23.980 INFO:teuthology.orchestra.run.smithi101.stdout:(7/7): buildah-1.21.4-2.module+el8.4.0+12050+ef 66 MB/s | 8.2 MB 00:00 2023-06-22T17:16:23.981 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-06-22T17:16:23.981 INFO:teuthology.orchestra.run.smithi101.stdout:Total 94 MB/s | 16 MB 00:00 2023-06-22T17:16:23.987 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-06-22T17:16:24.004 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-06-22T17:16:24.004 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-06-22T17:16:24.190 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-06-22T17:16:24.244 INFO:teuthology.orchestra.run.smithi039.stdout:========================================================================================================= 2023-06-22T17:16:24.244 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-06-22T17:16:24.244 INFO:teuthology.orchestra.run.smithi039.stdout:========================================================================================================= 2023-06-22T17:16:24.244 INFO:teuthology.orchestra.run.smithi039.stdout:Installing group/module packages: 2023-06-22T17:16:24.244 INFO:teuthology.orchestra.run.smithi039.stdout: buildah x86_64 1.21.4-2.module+el8.4.0+12050+ef972f71 rhel-8-for-x86_64-appstream-rpms 8.2 M 2023-06-22T17:16:24.245 INFO:teuthology.orchestra.run.smithi039.stdout: cockpit-podman noarch 32-2.module+el8.4.0+11990+22932769 rhel-8-for-x86_64-appstream-rpms 410 k 2023-06-22T17:16:24.245 INFO:teuthology.orchestra.run.smithi039.stdout: crun x86_64 0.20.1-1.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 192 k 2023-06-22T17:16:24.245 INFO:teuthology.orchestra.run.smithi039.stdout: skopeo x86_64 1:1.3.1-5.module+el8.4.0+11990+22932769 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-06-22T17:16:24.245 INFO:teuthology.orchestra.run.smithi039.stdout: toolbox noarch 0.0.8-1.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 16 k 2023-06-22T17:16:24.245 INFO:teuthology.orchestra.run.smithi039.stdout: udica noarch 0.2.4-2.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 51 k 2023-06-22T17:16:24.245 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2023-06-22T17:16:24.245 INFO:teuthology.orchestra.run.smithi039.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-06-22T17:16:24.245 INFO:teuthology.orchestra.run.smithi039.stdout:Installing module profiles: 2023-06-22T17:16:24.246 INFO:teuthology.orchestra.run.smithi039.stdout: container-tools/common 2023-06-22T17:16:24.246 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:16:24.246 INFO:teuthology.orchestra.run.smithi039.stdout:Enabling module streams: 2023-06-22T17:16:24.246 INFO:teuthology.orchestra.run.smithi039.stdout: container-tools rhel8 2023-06-22T17:16:24.246 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:16:24.246 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-06-22T17:16:24.246 INFO:teuthology.orchestra.run.smithi039.stdout:========================================================================================================= 2023-06-22T17:16:24.246 INFO:teuthology.orchestra.run.smithi039.stdout:Install 7 Packages 2023-06-22T17:16:24.247 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:16:24.247 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 16 M 2023-06-22T17:16:24.247 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 57 M 2023-06-22T17:16:24.247 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-06-22T17:16:24.280 INFO:teuthology.orchestra.run.smithi039.stdout:(1/7): yajl-2.1.0-10.el8.x86_64.rpm 1.2 MB/s | 41 kB 00:00 2023-06-22T17:16:24.290 INFO:teuthology.orchestra.run.smithi039.stdout:(2/7): crun-0.20.1-1.module+el8.4.0+11822+6cc1e 4.3 MB/s | 192 kB 00:00 2023-06-22T17:16:24.294 INFO:teuthology.orchestra.run.smithi039.stdout:(3/7): toolbox-0.0.8-1.module+el8.4.0+11822+6cc 1.1 MB/s | 16 kB 00:00 2023-06-22T17:16:24.300 INFO:teuthology.orchestra.run.smithi039.stdout:(4/7): cockpit-podman-32-2.module+el8.4.0+11990 7.4 MB/s | 410 kB 00:00 2023-06-22T17:16:24.317 INFO:teuthology.orchestra.run.smithi039.stdout:(5/7): udica-0.2.4-2.module+el8.4.0+11822+6cc1e 3.0 MB/s | 51 kB 00:00 2023-06-22T17:16:24.405 INFO:teuthology.orchestra.run.smithi039.stdout:(6/7): skopeo-1.3.1-5.module+el8.4.0+11990+2293 61 MB/s | 7.0 MB 00:00 2023-06-22T17:16:24.439 INFO:teuthology.orchestra.run.smithi039.stdout:(7/7): buildah-1.21.4-2.module+el8.4.0+12050+ef 57 MB/s | 8.2 MB 00:00 2023-06-22T17:16:24.439 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-06-22T17:16:24.440 INFO:teuthology.orchestra.run.smithi039.stdout:Total 82 MB/s | 16 MB 00:00 2023-06-22T17:16:24.446 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-06-22T17:16:24.450 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-06-22T17:16:24.454 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-06-22T17:16:24.465 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-06-22T17:16:24.465 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-06-22T17:16:24.551 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-06-22T17:16:24.555 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-06-22T17:16:24.979 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-06-22T17:16:24.983 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-06-22T17:16:25.003 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-06-22T17:16:25.008 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-06-22T17:16:25.160 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : yajl-2.1.0-10.el8.x86_64 1/7 2023-06-22T17:16:25.228 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : yajl-2.1.0-10.el8.x86_64 1/7 2023-06-22T17:16:25.342 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 2/7 2023-06-22T17:16:25.422 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 2/7 2023-06-22T17:16:25.423 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-06-22T17:16:25.647 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : yajl-2.1.0-10.el8.x86_64 1/7 2023-06-22T17:16:25.847 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 2/7 2023-06-22T17:16:26.273 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 3/7 2023-06-22T17:16:26.313 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 3/7 2023-06-22T17:16:26.778 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 3/7 2023-06-22T17:16:27.078 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_6 4/7 2023-06-22T17:16:27.119 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_6 4/7 2023-06-22T17:16:27.277 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_6 5/7 2023-06-22T17:16:27.315 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_6 5/7 2023-06-22T17:16:27.435 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 6/7 2023-06-22T17:16:27.464 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 6/7 2023-06-22T17:16:27.591 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_6 4/7 2023-06-22T17:16:27.598 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : cockpit-podman-32-2.module+el8.4.0+11990+22932769.no 7/7 2023-06-22T17:16:27.619 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : cockpit-podman-32-2.module+el8.4.0+11990+22932769.no 7/7 2023-06-22T17:16:27.805 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_6 5/7 2023-06-22T17:16:27.875 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: cockpit-podman-32-2.module+el8.4.0+11990+22932769.no 7/7 2023-06-22T17:16:27.875 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 1/7 2023-06-22T17:16:27.875 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 2/7 2023-06-22T17:16:27.875 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : cockpit-podman-32-2.module+el8.4.0+11990+22932769.no 3/7 2023-06-22T17:16:27.876 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 4/7 2023-06-22T17:16:27.876 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_6 5/7 2023-06-22T17:16:27.876 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_6 6/7 2023-06-22T17:16:27.889 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: cockpit-podman-32-2.module+el8.4.0+11990+22932769.no 7/7 2023-06-22T17:16:27.889 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 1/7 2023-06-22T17:16:27.889 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 2/7 2023-06-22T17:16:27.889 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : cockpit-podman-32-2.module+el8.4.0+11990+22932769.no 3/7 2023-06-22T17:16:27.889 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 4/7 2023-06-22T17:16:27.889 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_6 5/7 2023-06-22T17:16:27.890 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_6 6/7 2023-06-22T17:16:27.990 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 6/7 2023-06-22T17:16:28.151 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cockpit-podman-32-2.module+el8.4.0+11990+22932769.no 7/7 2023-06-22T17:16:28.422 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: cockpit-podman-32-2.module+el8.4.0+11990+22932769.no 7/7 2023-06-22T17:16:28.422 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 1/7 2023-06-22T17:16:28.422 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 2/7 2023-06-22T17:16:28.422 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cockpit-podman-32-2.module+el8.4.0+11990+22932769.no 3/7 2023-06-22T17:16:28.422 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 4/7 2023-06-22T17:16:28.422 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_6 5/7 2023-06-22T17:16:28.423 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_6 6/7 2023-06-22T17:16:28.917 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 7/7 2023-06-22T17:16:28.917 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:48 ago on Thu 22 Jun 2023 05:14:40 PM UTC. 2023-06-22T17:16:29.332 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 7/7 2023-06-22T17:16:29.332 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:01:51 ago on Thu 22 Jun 2023 05:14:37 PM UTC. 2023-06-22T17:16:29.617 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 7/7 2023-06-22T17:16:29.617 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:40 ago on Thu 22 Jun 2023 05:14:49 PM UTC. 2023-06-22T17:16:30.931 INFO:teuthology.orchestra.run.smithi133.stdout:Installed products updated. 2023-06-22T17:16:31.037 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-06-22T17:16:31.037 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-06-22T17:16:31.037 INFO:teuthology.orchestra.run.smithi133.stdout: buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_64 2023-06-22T17:16:31.038 INFO:teuthology.orchestra.run.smithi133.stdout: cockpit-podman-32-2.module+el8.4.0+11990+22932769.noarch 2023-06-22T17:16:31.038 INFO:teuthology.orchestra.run.smithi133.stdout: crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 2023-06-22T17:16:31.038 INFO:teuthology.orchestra.run.smithi133.stdout: skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_64 2023-06-22T17:16:31.038 INFO:teuthology.orchestra.run.smithi133.stdout: toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 2023-06-22T17:16:31.038 INFO:teuthology.orchestra.run.smithi133.stdout: udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 2023-06-22T17:16:31.038 INFO:teuthology.orchestra.run.smithi133.stdout: yajl-2.1.0-10.el8.x86_64 2023-06-22T17:16:31.038 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-06-22T17:16:31.039 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-06-22T17:16:31.165 DEBUG:teuthology.parallel:result is None 2023-06-22T17:16:31.332 INFO:teuthology.orchestra.run.smithi101.stdout:Installed products updated. 2023-06-22T17:16:31.437 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-06-22T17:16:31.437 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-06-22T17:16:31.437 INFO:teuthology.orchestra.run.smithi101.stdout: buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_64 2023-06-22T17:16:31.437 INFO:teuthology.orchestra.run.smithi101.stdout: cockpit-podman-32-2.module+el8.4.0+11990+22932769.noarch 2023-06-22T17:16:31.438 INFO:teuthology.orchestra.run.smithi101.stdout: crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 2023-06-22T17:16:31.438 INFO:teuthology.orchestra.run.smithi101.stdout: skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_64 2023-06-22T17:16:31.438 INFO:teuthology.orchestra.run.smithi101.stdout: toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 2023-06-22T17:16:31.438 INFO:teuthology.orchestra.run.smithi101.stdout: udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 2023-06-22T17:16:31.439 INFO:teuthology.orchestra.run.smithi101.stdout: yajl-2.1.0-10.el8.x86_64 2023-06-22T17:16:31.439 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-06-22T17:16:31.439 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-06-22T17:16:31.558 DEBUG:teuthology.parallel:result is None 2023-06-22T17:16:31.657 INFO:teuthology.orchestra.run.smithi039.stdout:Installed products updated. 2023-06-22T17:16:31.776 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:16:31.777 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-06-22T17:16:31.777 INFO:teuthology.orchestra.run.smithi039.stdout: buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_64 2023-06-22T17:16:31.777 INFO:teuthology.orchestra.run.smithi039.stdout: cockpit-podman-32-2.module+el8.4.0+11990+22932769.noarch 2023-06-22T17:16:31.777 INFO:teuthology.orchestra.run.smithi039.stdout: crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 2023-06-22T17:16:31.777 INFO:teuthology.orchestra.run.smithi039.stdout: skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_64 2023-06-22T17:16:31.778 INFO:teuthology.orchestra.run.smithi039.stdout: toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 2023-06-22T17:16:31.778 INFO:teuthology.orchestra.run.smithi039.stdout: udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 2023-06-22T17:16:31.778 INFO:teuthology.orchestra.run.smithi039.stdout: yajl-2.1.0-10.el8.x86_64 2023-06-22T17:16:31.778 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:16:31.778 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-06-22T17:16:31.899 DEBUG:teuthology.parallel:result is None 2023-06-22T17:16:31.899 INFO:teuthology.run_tasks:Running task install... 2023-06-22T17:16:31.909 DEBUG:teuthology.task.install:project ceph 2023-06-22T17:16:31.910 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'}} 2023-06-22T17:16:31.910 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'} 2023-06-22T17:16:31.910 INFO:teuthology.task.install:Using flavor: default 2023-06-22T17:16:31.916 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-06-22T17:16:31.917 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-06-22T17:16:31.917 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c', 'tag': None, 'wait_for_package': False} 2023-06-22T17:16:31.918 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-22T17:16:31.920 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c', 'tag': None, 'wait_for_package': False} 2023-06-22T17:16:31.921 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-22T17:16:31.922 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c', 'tag': None, 'wait_for_package': False} 2023-06-22T17:16:31.922 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-22T17:16:32.069 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/ 2023-06-22T17:16:32.069 INFO:teuthology.task.install.rpm:Package version is 17.2.6-719.g42ddc46e 2023-06-22T17:16:32.173 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-06-22T17:16:32.173 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-06-22T17:16:32.174 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-06-22T17:16:32.182 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/ 2023-06-22T17:16:32.182 INFO:teuthology.task.install.rpm:Package version is 17.2.6-719.g42ddc46e 2023-06-22T17:16:32.186 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/ 2023-06-22T17:16:32.186 INFO:teuthology.task.install.rpm:Package version is 17.2.6-719.g42ddc46e 2023-06-22T17:16:32.201 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2023-06-22T17:16:32.201 DEBUG:teuthology.orchestra.run.smithi101:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/42ddc46e6e6582d8e29a5a982a81886fe028be2c/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-06-22T17:16:32.268 DEBUG:teuthology.orchestra.run.smithi101:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-06-22T17:16:32.289 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-06-22T17:16:32.290 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-22T17:16:32.290 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-06-22T17:16:32.316 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2023-06-22T17:16:32.317 DEBUG:teuthology.orchestra.run.smithi039:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/42ddc46e6e6582d8e29a5a982a81886fe028be2c/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-06-22T17:16:32.345 DEBUG:teuthology.orchestra.run.smithi101:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-06-22T17:16:32.368 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-06-22T17:16:32.368 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-06-22T17:16:32.368 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-06-22T17:16:32.383 DEBUG:teuthology.orchestra.run.smithi039:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-06-22T17:16:32.395 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2023-06-22T17:16:32.395 DEBUG:teuthology.orchestra.run.smithi133:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/42ddc46e6e6582d8e29a5a982a81886fe028be2c/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-06-22T17:16:32.409 INFO:teuthology.orchestra.run.smithi101.stdout:check_obsoletes = 1 2023-06-22T17:16:32.411 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum clean all 2023-06-22T17:16:32.459 DEBUG:teuthology.orchestra.run.smithi039:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-06-22T17:16:32.461 DEBUG:teuthology.orchestra.run.smithi133:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-06-22T17:16:32.523 INFO:teuthology.orchestra.run.smithi039.stdout:check_obsoletes = 1 2023-06-22T17:16:32.525 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum clean all 2023-06-22T17:16:32.537 DEBUG:teuthology.orchestra.run.smithi133:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-06-22T17:16:32.601 INFO:teuthology.orchestra.run.smithi133.stdout:check_obsoletes = 1 2023-06-22T17:16:32.603 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum clean all 2023-06-22T17:16:32.801 INFO:teuthology.orchestra.run.smithi101.stdout:Updating Subscription Management repositories. 2023-06-22T17:16:32.934 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-06-22T17:16:32.991 INFO:teuthology.orchestra.run.smithi133.stdout:Updating Subscription Management repositories. 2023-06-22T17:16:33.135 INFO:teuthology.orchestra.run.smithi101.stdout:44 files removed 2023-06-22T17:16:33.186 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2023-06-22T17:16:33.237 INFO:teuthology.orchestra.run.smithi039.stdout:44 files removed 2023-06-22T17:16:33.294 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2023-06-22T17:16:33.302 INFO:teuthology.orchestra.run.smithi133.stdout:44 files removed 2023-06-22T17:16:33.352 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2023-06-22T17:16:33.534 INFO:teuthology.orchestra.run.smithi101.stdout:Updating Subscription Management repositories. 2023-06-22T17:16:33.662 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-06-22T17:16:33.702 INFO:teuthology.orchestra.run.smithi133.stdout:Updating Subscription Management repositories. 2023-06-22T17:16:34.414 INFO:teuthology.orchestra.run.smithi101.stdout:ceph packages for x86_64 181 kB/s | 78 kB 00:00 2023-06-22T17:16:34.495 INFO:teuthology.orchestra.run.smithi133.stdout:ceph packages for x86_64 245 kB/s | 78 kB 00:00 2023-06-22T17:16:34.534 INFO:teuthology.orchestra.run.smithi039.stdout:ceph packages for x86_64 197 kB/s | 78 kB 00:00 2023-06-22T17:16:34.702 INFO:teuthology.orchestra.run.smithi101.stdout:ceph noarch packages 37 kB/s | 9.2 kB 00:00 2023-06-22T17:16:34.792 INFO:teuthology.orchestra.run.smithi133.stdout:ceph noarch packages 36 kB/s | 9.2 kB 00:00 2023-06-22T17:16:34.895 INFO:teuthology.orchestra.run.smithi039.stdout:ceph noarch packages 29 kB/s | 9.2 kB 00:00 2023-06-22T17:16:35.041 INFO:teuthology.orchestra.run.smithi101.stdout:ceph source packages 4.7 kB/s | 1.4 kB 00:00 2023-06-22T17:16:35.064 INFO:teuthology.orchestra.run.smithi133.stdout:ceph source packages 6.0 kB/s | 1.4 kB 00:00 2023-06-22T17:16:35.168 INFO:teuthology.orchestra.run.smithi039.stdout:ceph source packages 6.0 kB/s | 1.4 kB 00:00 2023-06-22T17:16:37.584 INFO:teuthology.orchestra.run.smithi133.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2023-06-22T17:16:37.585 INFO:teuthology.orchestra.run.smithi101.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2023-06-22T17:16:37.669 INFO:teuthology.orchestra.run.smithi039.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2023-06-22T17:16:39.093 INFO:teuthology.orchestra.run.smithi133.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2023-06-22T17:16:40.453 INFO:teuthology.orchestra.run.smithi101.stdout:Extra Packages for Enterprise Linux 5.5 MB/s | 16 MB 00:02 2023-06-22T17:16:42.239 INFO:teuthology.orchestra.run.smithi039.stdout:Extra Packages for Enterprise Linux 3.4 MB/s | 16 MB 00:04 2023-06-22T17:16:42.368 INFO:teuthology.orchestra.run.smithi133.stdout:lab-extras 1.7 MB/s | 24 kB 00:00 2023-06-22T17:16:42.749 INFO:teuthology.orchestra.run.smithi133.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 127 MB/s | 38 MB 00:00 2023-06-22T17:16:43.699 INFO:teuthology.orchestra.run.smithi101.stdout:lab-extras 4.2 MB/s | 24 kB 00:00 2023-06-22T17:16:44.093 INFO:teuthology.orchestra.run.smithi101.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 105 MB/s | 33 MB 00:00 2023-06-22T17:16:45.713 INFO:teuthology.orchestra.run.smithi039.stdout:lab-extras 4.2 MB/s | 24 kB 00:00 2023-06-22T17:16:46.105 INFO:teuthology.orchestra.run.smithi039.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 108 MB/s | 33 MB 00:00 2023-06-22T17:16:48.249 INFO:teuthology.orchestra.run.smithi133.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 123 MB/s | 33 MB 00:00 2023-06-22T17:16:49.197 INFO:teuthology.orchestra.run.smithi101.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 127 MB/s | 38 MB 00:00 2023-06-22T17:16:51.559 INFO:teuthology.orchestra.run.smithi039.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 112 MB/s | 38 MB 00:00 2023-06-22T17:16:56.135 INFO:teuthology.orchestra.run.smithi133.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-22T17:16:56.135 INFO:teuthology.orchestra.run.smithi133.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-22T17:16:56.146 INFO:teuthology.orchestra.run.smithi133.stdout:Package NetworkManager-1:1.30.0-13.el8_4.x86_64 is already installed. 2023-06-22T17:16:56.147 INFO:teuthology.orchestra.run.smithi133.stdout:Package iproute-5.9.0-4.el8.x86_64 is already installed. 2023-06-22T17:16:56.147 INFO:teuthology.orchestra.run.smithi133.stdout:Package util-linux-2.32.1-27.el8.x86_64 is already installed. 2023-06-22T17:16:56.148 INFO:teuthology.orchestra.run.smithi133.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-06-22T17:16:56.150 INFO:teuthology.orchestra.run.smithi133.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-06-22T17:16:56.152 INFO:teuthology.orchestra.run.smithi133.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2023-06-22T17:16:56.153 INFO:teuthology.orchestra.run.smithi133.stdout:Package xfsprogs-5.0.0-8.el8.x86_64 is already installed. 2023-06-22T17:16:56.637 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-06-22T17:16:56.648 INFO:teuthology.orchestra.run.smithi133.stdout:====================================================================================================================== 2023-06-22T17:16:56.648 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2023-06-22T17:16:56.648 INFO:teuthology.orchestra.run.smithi133.stdout:====================================================================================================================== 2023-06-22T17:16:56.648 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-06-22T17:16:56.649 INFO:teuthology.orchestra.run.smithi133.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-06-22T17:16:56.649 INFO:teuthology.orchestra.run.smithi133.stdout: ceph x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 6.4 k 2023-06-22T17:16:56.649 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-base x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.2 M 2023-06-22T17:16:56.649 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 873 k 2023-06-22T17:16:56.649 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 162 k 2023-06-22T17:16:56.649 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 1.6 M 2023-06-22T17:16:56.649 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 121 k 2023-06-22T17:16:56.650 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-dashboard noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 1.7 M 2023-06-22T17:16:56.650 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 7.4 M 2023-06-22T17:16:56.650 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-rook noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 47 k 2023-06-22T17:16:56.650 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-radosgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-22T17:16:56.650 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-test x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 52 M 2023-06-22T17:16:56.650 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-volume noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 264 k 2023-06-22T17:16:56.650 INFO:teuthology.orchestra.run.smithi133.stdout: cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 80 k 2023-06-22T17:16:56.651 INFO:teuthology.orchestra.run.smithi133.stdout: cephfs-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 255 k 2023-06-22T17:16:56.651 INFO:teuthology.orchestra.run.smithi133.stdout: cephfs-top noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 18 k 2023-06-22T17:16:56.651 INFO:teuthology.orchestra.run.smithi133.stdout: elfutils-libelf-devel x86_64 0.182-3.el8 rhel-8-for-x86_64-baseos-rpms 59 k 2023-06-22T17:16:56.651 INFO:teuthology.orchestra.run.smithi133.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-06-22T17:16:56.651 INFO:teuthology.orchestra.run.smithi133.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-06-22T17:16:56.651 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 25 k 2023-06-22T17:16:56.651 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 794 k 2023-06-22T17:16:56.651 INFO:teuthology.orchestra.run.smithi133.stdout: librados-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 130 k 2023-06-22T17:16:56.652 INFO:teuthology.orchestra.run.smithi133.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-06-22T17:16:56.652 INFO:teuthology.orchestra.run.smithi133.stdout: libuuid-devel x86_64 2.32.1-27.el8 rhel-8-for-x86_64-baseos-rpms 97 k 2023-06-22T17:16:56.652 INFO:teuthology.orchestra.run.smithi133.stdout: openssl-devel x86_64 1:1.1.1g-15.el8_3 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-06-22T17:16:56.652 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cephfs x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 190 k 2023-06-22T17:16:56.652 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rados x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 371 k 2023-06-22T17:16:56.652 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 357 k 2023-06-22T17:16:56.652 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 107 k 2023-06-22T17:16:56.653 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 91 k 2023-06-22T17:16:56.653 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-22T17:16:56.653 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-nbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 173 k 2023-06-22T17:16:56.653 INFO:teuthology.orchestra.run.smithi133.stdout: xfsprogs-devel x86_64 5.0.0-8.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2023-06-22T17:16:56.653 INFO:teuthology.orchestra.run.smithi133.stdout:Upgrading: 2023-06-22T17:16:56.653 INFO:teuthology.orchestra.run.smithi133.stdout: librados2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-22T17:16:56.653 INFO:teuthology.orchestra.run.smithi133.stdout: librbd1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.6 M 2023-06-22T17:16:56.653 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2023-06-22T17:16:56.654 INFO:teuthology.orchestra.run.smithi133.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-06-22T17:16:56.654 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 23 M 2023-06-22T17:16:56.654 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-grafana-dashboards noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 23 k 2023-06-22T17:16:56.654 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mds x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 2.3 M 2023-06-22T17:16:56.654 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-modules-core noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 238 k 2023-06-22T17:16:56.654 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mon x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.9 M 2023-06-22T17:16:56.654 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-osd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-22T17:16:56.655 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-prometheus-alerts noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 14 k 2023-06-22T17:16:56.655 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-selinux x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 24 k 2023-06-22T17:16:56.655 INFO:teuthology.orchestra.run.smithi133.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-06-22T17:16:56.655 INFO:teuthology.orchestra.run.smithi133.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-06-22T17:16:56.655 INFO:teuthology.orchestra.run.smithi133.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-06-22T17:16:56.655 INFO:teuthology.orchestra.run.smithi133.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-06-22T17:16:56.655 INFO:teuthology.orchestra.run.smithi133.stdout: krb5-devel x86_64 1.18.2-8.3.el8_4 rhel-8-for-x86_64-baseos-rpms 559 k 2023-06-22T17:16:56.656 INFO:teuthology.orchestra.run.smithi133.stdout: libcephsqlite x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 174 k 2023-06-22T17:16:56.656 INFO:teuthology.orchestra.run.smithi133.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2023-06-22T17:16:56.656 INFO:teuthology.orchestra.run.smithi133.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-06-22T17:16:56.656 INFO:teuthology.orchestra.run.smithi133.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-06-22T17:16:56.656 INFO:teuthology.orchestra.run.smithi133.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-06-22T17:16:56.656 INFO:teuthology.orchestra.run.smithi133.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-06-22T17:16:56.656 INFO:teuthology.orchestra.run.smithi133.stdout: libradosstriper1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 506 k 2023-06-22T17:16:56.656 INFO:teuthology.orchestra.run.smithi133.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-06-22T17:16:56.657 INFO:teuthology.orchestra.run.smithi133.stdout: librgw2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.5 M 2023-06-22T17:16:56.657 INFO:teuthology.orchestra.run.smithi133.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-06-22T17:16:56.657 INFO:teuthology.orchestra.run.smithi133.stdout: libsepol-devel x86_64 2.9-2.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2023-06-22T17:16:56.657 INFO:teuthology.orchestra.run.smithi133.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-06-22T17:16:56.657 INFO:teuthology.orchestra.run.smithi133.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-06-22T17:16:56.657 INFO:teuthology.orchestra.run.smithi133.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-06-22T17:16:56.657 INFO:teuthology.orchestra.run.smithi133.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-06-22T17:16:56.657 INFO:teuthology.orchestra.run.smithi133.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-06-22T17:16:56.658 INFO:teuthology.orchestra.run.smithi133.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-06-22T17:16:56.658 INFO:teuthology.orchestra.run.smithi133.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-06-22T17:16:56.658 INFO:teuthology.orchestra.run.smithi133.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-06-22T17:16:56.658 INFO:teuthology.orchestra.run.smithi133.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-06-22T17:16:56.658 INFO:teuthology.orchestra.run.smithi133.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-06-22T17:16:56.658 INFO:teuthology.orchestra.run.smithi133.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-06-22T17:16:56.658 INFO:teuthology.orchestra.run.smithi133.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-06-22T17:16:56.659 INFO:teuthology.orchestra.run.smithi133.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-06-22T17:16:56.659 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-06-22T17:16:56.659 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-argparse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 45 k 2023-06-22T17:16:56.659 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 115 k 2023-06-22T17:16:56.659 INFO:teuthology.orchestra.run.smithi133.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-06-22T17:16:56.659 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-06-22T17:16:56.660 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-06-22T17:16:56.660 INFO:teuthology.orchestra.run.smithi133.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-06-22T17:16:56.660 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-06-22T17:16:56.660 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-06-22T17:16:56.660 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-06-22T17:16:56.660 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-06-22T17:16:56.660 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-06-22T17:16:56.660 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-22T17:16:56.661 INFO:teuthology.orchestra.run.smithi133.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-06-22T17:16:56.661 INFO:teuthology.orchestra.run.smithi133.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-06-22T17:16:56.661 INFO:teuthology.orchestra.run.smithi133.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-06-22T17:16:56.661 INFO:teuthology.orchestra.run.smithi133.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-06-22T17:16:56.661 INFO:teuthology.orchestra.run.smithi133.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-06-22T17:16:56.661 INFO:teuthology.orchestra.run.smithi133.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-06-22T17:16:56.661 INFO:teuthology.orchestra.run.smithi133.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-06-22T17:16:56.662 INFO:teuthology.orchestra.run.smithi133.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-06-22T17:16:56.662 INFO:teuthology.orchestra.run.smithi133.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-06-22T17:16:56.662 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-06-22T17:16:56.662 INFO:teuthology.orchestra.run.smithi133.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-06-22T17:16:56.662 INFO:teuthology.orchestra.run.smithi133.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-06-22T17:16:56.662 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-06-22T17:16:56.662 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-06-22T17:16:56.662 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-06-22T17:16:56.663 INFO:teuthology.orchestra.run.smithi133.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-06-22T17:16:56.663 INFO:teuthology.orchestra.run.smithi133.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-22T17:16:56.663 INFO:teuthology.orchestra.run.smithi133.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-06-22T17:16:56.663 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-06-22T17:16:56.663 INFO:teuthology.orchestra.run.smithi133.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-06-22T17:16:56.663 INFO:teuthology.orchestra.run.smithi133.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-06-22T17:16:56.663 INFO:teuthology.orchestra.run.smithi133.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-06-22T17:16:56.664 INFO:teuthology.orchestra.run.smithi133.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-06-22T17:16:56.664 INFO:teuthology.orchestra.run.smithi133.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-06-22T17:16:56.664 INFO:teuthology.orchestra.run.smithi133.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-06-22T17:16:56.664 INFO:teuthology.orchestra.run.smithi133.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-06-22T17:16:56.664 INFO:teuthology.orchestra.run.smithi133.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-06-22T17:16:56.664 INFO:teuthology.orchestra.run.smithi133.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-06-22T17:16:56.664 INFO:teuthology.orchestra.run.smithi133.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-06-22T17:16:56.664 INFO:teuthology.orchestra.run.smithi133.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-06-22T17:16:56.665 INFO:teuthology.orchestra.run.smithi133.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-06-22T17:16:56.665 INFO:teuthology.orchestra.run.smithi133.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-06-22T17:16:56.665 INFO:teuthology.orchestra.run.smithi133.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-06-22T17:16:56.665 INFO:teuthology.orchestra.run.smithi133.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2023-06-22T17:16:56.665 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-06-22T17:16:56.665 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-06-22T17:16:56.665 INFO:teuthology.orchestra.run.smithi133.stdout:====================================================================================================================== 2023-06-22T17:16:56.665 INFO:teuthology.orchestra.run.smithi133.stdout:Install 118 Packages 2023-06-22T17:16:56.666 INFO:teuthology.orchestra.run.smithi133.stdout:Upgrade 2 Packages 2023-06-22T17:16:56.666 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-06-22T17:16:56.666 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 201 M 2023-06-22T17:16:56.666 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-06-22T17:16:57.190 INFO:teuthology.orchestra.run.smithi133.stdout:(1/120): ceph-17.2.6-719.g42ddc46e.el8.x86_64.r 43 kB/s | 6.4 kB 00:00 2023-06-22T17:16:57.471 INFO:teuthology.orchestra.run.smithi133.stdout:(2/120): ceph-fuse-17.2.6-719.g42ddc46e.el8.x86 3.0 MB/s | 873 kB 00:00 2023-06-22T17:16:57.526 INFO:teuthology.orchestra.run.smithi133.stdout:(3/120): ceph-immutable-object-cache-17.2.6-719 2.9 MB/s | 162 kB 00:00 2023-06-22T17:16:57.565 INFO:teuthology.orchestra.run.smithi101.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-22T17:16:57.566 INFO:teuthology.orchestra.run.smithi101.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-22T17:16:57.576 INFO:teuthology.orchestra.run.smithi101.stdout:Package NetworkManager-1:1.30.0-13.el8_4.x86_64 is already installed. 2023-06-22T17:16:57.577 INFO:teuthology.orchestra.run.smithi101.stdout:Package iproute-5.9.0-4.el8.x86_64 is already installed. 2023-06-22T17:16:57.578 INFO:teuthology.orchestra.run.smithi101.stdout:Package util-linux-2.32.1-27.el8.x86_64 is already installed. 2023-06-22T17:16:57.579 INFO:teuthology.orchestra.run.smithi101.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-06-22T17:16:57.581 INFO:teuthology.orchestra.run.smithi101.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-06-22T17:16:57.583 INFO:teuthology.orchestra.run.smithi101.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2023-06-22T17:16:57.584 INFO:teuthology.orchestra.run.smithi101.stdout:Package xfsprogs-5.0.0-8.el8.x86_64 is already installed. 2023-06-22T17:16:57.874 INFO:teuthology.orchestra.run.smithi133.stdout:(4/120): ceph-mds-17.2.6-719.g42ddc46e.el8.x86_ 6.6 MB/s | 2.3 MB 00:00 2023-06-22T17:16:58.071 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-06-22T17:16:58.075 INFO:teuthology.orchestra.run.smithi133.stdout:(5/120): ceph-base-17.2.6-719.g42ddc46e.el8.x86 5.0 MB/s | 5.2 MB 00:01 2023-06-22T17:16:58.082 INFO:teuthology.orchestra.run.smithi101.stdout:====================================================================================================================== 2023-06-22T17:16:58.082 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repository Size 2023-06-22T17:16:58.083 INFO:teuthology.orchestra.run.smithi101.stdout:====================================================================================================================== 2023-06-22T17:16:58.083 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-06-22T17:16:58.083 INFO:teuthology.orchestra.run.smithi101.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-06-22T17:16:58.083 INFO:teuthology.orchestra.run.smithi101.stdout: ceph x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 6.4 k 2023-06-22T17:16:58.083 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-base x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.2 M 2023-06-22T17:16:58.083 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 873 k 2023-06-22T17:16:58.083 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 162 k 2023-06-22T17:16:58.083 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 1.6 M 2023-06-22T17:16:58.084 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 121 k 2023-06-22T17:16:58.084 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-dashboard noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 1.7 M 2023-06-22T17:16:58.084 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 7.4 M 2023-06-22T17:16:58.084 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-rook noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 47 k 2023-06-22T17:16:58.084 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-radosgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-22T17:16:58.084 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-test x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 52 M 2023-06-22T17:16:58.084 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-volume noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 264 k 2023-06-22T17:16:58.085 INFO:teuthology.orchestra.run.smithi101.stdout: cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 80 k 2023-06-22T17:16:58.085 INFO:teuthology.orchestra.run.smithi101.stdout: cephfs-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 255 k 2023-06-22T17:16:58.085 INFO:teuthology.orchestra.run.smithi101.stdout: cephfs-top noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 18 k 2023-06-22T17:16:58.085 INFO:teuthology.orchestra.run.smithi101.stdout: elfutils-libelf-devel x86_64 0.182-3.el8 rhel-8-for-x86_64-baseos-rpms 59 k 2023-06-22T17:16:58.085 INFO:teuthology.orchestra.run.smithi101.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-06-22T17:16:58.085 INFO:teuthology.orchestra.run.smithi101.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-06-22T17:16:58.086 INFO:teuthology.orchestra.run.smithi101.stdout: libcephfs-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 25 k 2023-06-22T17:16:58.086 INFO:teuthology.orchestra.run.smithi101.stdout: libcephfs2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 794 k 2023-06-22T17:16:58.086 INFO:teuthology.orchestra.run.smithi101.stdout: librados-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 130 k 2023-06-22T17:16:58.086 INFO:teuthology.orchestra.run.smithi101.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-06-22T17:16:58.086 INFO:teuthology.orchestra.run.smithi101.stdout: libuuid-devel x86_64 2.32.1-27.el8 rhel-8-for-x86_64-baseos-rpms 97 k 2023-06-22T17:16:58.086 INFO:teuthology.orchestra.run.smithi101.stdout: openssl-devel x86_64 1:1.1.1g-15.el8_3 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-06-22T17:16:58.087 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cephfs x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 190 k 2023-06-22T17:16:58.087 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rados x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 371 k 2023-06-22T17:16:58.087 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 357 k 2023-06-22T17:16:58.087 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 107 k 2023-06-22T17:16:58.087 INFO:teuthology.orchestra.run.smithi101.stdout: rbd-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 91 k 2023-06-22T17:16:58.087 INFO:teuthology.orchestra.run.smithi101.stdout: rbd-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-22T17:16:58.087 INFO:teuthology.orchestra.run.smithi101.stdout: rbd-nbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 173 k 2023-06-22T17:16:58.088 INFO:teuthology.orchestra.run.smithi101.stdout: xfsprogs-devel x86_64 5.0.0-8.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2023-06-22T17:16:58.088 INFO:teuthology.orchestra.run.smithi101.stdout:Upgrading: 2023-06-22T17:16:58.088 INFO:teuthology.orchestra.run.smithi101.stdout: librados2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-22T17:16:58.088 INFO:teuthology.orchestra.run.smithi101.stdout: librbd1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.6 M 2023-06-22T17:16:58.088 INFO:teuthology.orchestra.run.smithi101.stdout:Installing dependencies: 2023-06-22T17:16:58.088 INFO:teuthology.orchestra.run.smithi101.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-06-22T17:16:58.089 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 23 M 2023-06-22T17:16:58.089 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-grafana-dashboards noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 23 k 2023-06-22T17:16:58.089 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mds x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 2.3 M 2023-06-22T17:16:58.089 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-modules-core noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 238 k 2023-06-22T17:16:58.089 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mon x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.9 M 2023-06-22T17:16:58.089 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-osd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-22T17:16:58.090 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-prometheus-alerts noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 14 k 2023-06-22T17:16:58.090 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-selinux x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 24 k 2023-06-22T17:16:58.090 INFO:teuthology.orchestra.run.smithi101.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-06-22T17:16:58.090 INFO:teuthology.orchestra.run.smithi101.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-06-22T17:16:58.090 INFO:teuthology.orchestra.run.smithi101.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-06-22T17:16:58.090 INFO:teuthology.orchestra.run.smithi101.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-06-22T17:16:58.091 INFO:teuthology.orchestra.run.smithi101.stdout: krb5-devel x86_64 1.18.2-8.3.el8_4 rhel-8-for-x86_64-baseos-rpms 559 k 2023-06-22T17:16:58.091 INFO:teuthology.orchestra.run.smithi101.stdout: libcephsqlite x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 174 k 2023-06-22T17:16:58.091 INFO:teuthology.orchestra.run.smithi101.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2023-06-22T17:16:58.091 INFO:teuthology.orchestra.run.smithi101.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-06-22T17:16:58.091 INFO:teuthology.orchestra.run.smithi101.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-06-22T17:16:58.091 INFO:teuthology.orchestra.run.smithi101.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-06-22T17:16:58.092 INFO:teuthology.orchestra.run.smithi101.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-06-22T17:16:58.092 INFO:teuthology.orchestra.run.smithi101.stdout: libradosstriper1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 506 k 2023-06-22T17:16:58.092 INFO:teuthology.orchestra.run.smithi101.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-06-22T17:16:58.092 INFO:teuthology.orchestra.run.smithi101.stdout: librgw2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.5 M 2023-06-22T17:16:58.092 INFO:teuthology.orchestra.run.smithi101.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-06-22T17:16:58.092 INFO:teuthology.orchestra.run.smithi101.stdout: libsepol-devel x86_64 2.9-2.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2023-06-22T17:16:58.092 INFO:teuthology.orchestra.run.smithi101.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-06-22T17:16:58.093 INFO:teuthology.orchestra.run.smithi101.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-06-22T17:16:58.093 INFO:teuthology.orchestra.run.smithi101.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-06-22T17:16:58.093 INFO:teuthology.orchestra.run.smithi101.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-06-22T17:16:58.093 INFO:teuthology.orchestra.run.smithi101.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-06-22T17:16:58.093 INFO:teuthology.orchestra.run.smithi101.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-06-22T17:16:58.093 INFO:teuthology.orchestra.run.smithi101.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-06-22T17:16:58.093 INFO:teuthology.orchestra.run.smithi101.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-06-22T17:16:58.093 INFO:teuthology.orchestra.run.smithi101.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-06-22T17:16:58.094 INFO:teuthology.orchestra.run.smithi101.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-06-22T17:16:58.094 INFO:teuthology.orchestra.run.smithi101.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-06-22T17:16:58.094 INFO:teuthology.orchestra.run.smithi101.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-06-22T17:16:58.094 INFO:teuthology.orchestra.run.smithi101.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-06-22T17:16:58.094 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-06-22T17:16:58.094 INFO:teuthology.orchestra.run.smithi101.stdout: python3-ceph-argparse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 45 k 2023-06-22T17:16:58.094 INFO:teuthology.orchestra.run.smithi101.stdout: python3-ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 115 k 2023-06-22T17:16:58.095 INFO:teuthology.orchestra.run.smithi101.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-06-22T17:16:58.095 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-06-22T17:16:58.095 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-06-22T17:16:58.095 INFO:teuthology.orchestra.run.smithi101.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-06-22T17:16:58.095 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-06-22T17:16:58.095 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-06-22T17:16:58.095 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-06-22T17:16:58.095 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-06-22T17:16:58.096 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-06-22T17:16:58.096 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-22T17:16:58.096 INFO:teuthology.orchestra.run.smithi101.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-06-22T17:16:58.096 INFO:teuthology.orchestra.run.smithi101.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-06-22T17:16:58.096 INFO:teuthology.orchestra.run.smithi101.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-06-22T17:16:58.096 INFO:teuthology.orchestra.run.smithi101.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-06-22T17:16:58.096 INFO:teuthology.orchestra.run.smithi101.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-06-22T17:16:58.096 INFO:teuthology.orchestra.run.smithi101.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-06-22T17:16:58.097 INFO:teuthology.orchestra.run.smithi101.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-06-22T17:16:58.097 INFO:teuthology.orchestra.run.smithi101.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-06-22T17:16:58.097 INFO:teuthology.orchestra.run.smithi101.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-06-22T17:16:58.097 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-06-22T17:16:58.097 INFO:teuthology.orchestra.run.smithi101.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-06-22T17:16:58.097 INFO:teuthology.orchestra.run.smithi101.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-06-22T17:16:58.097 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-06-22T17:16:58.097 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-06-22T17:16:58.098 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-06-22T17:16:58.098 INFO:teuthology.orchestra.run.smithi101.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-06-22T17:16:58.098 INFO:teuthology.orchestra.run.smithi101.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-22T17:16:58.098 INFO:teuthology.orchestra.run.smithi101.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-06-22T17:16:58.098 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-06-22T17:16:58.098 INFO:teuthology.orchestra.run.smithi101.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-06-22T17:16:58.098 INFO:teuthology.orchestra.run.smithi101.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-06-22T17:16:58.099 INFO:teuthology.orchestra.run.smithi101.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-06-22T17:16:58.099 INFO:teuthology.orchestra.run.smithi101.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-06-22T17:16:58.099 INFO:teuthology.orchestra.run.smithi101.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-06-22T17:16:58.099 INFO:teuthology.orchestra.run.smithi101.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-06-22T17:16:58.099 INFO:teuthology.orchestra.run.smithi101.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-06-22T17:16:58.099 INFO:teuthology.orchestra.run.smithi101.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-06-22T17:16:58.099 INFO:teuthology.orchestra.run.smithi101.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-06-22T17:16:58.099 INFO:teuthology.orchestra.run.smithi101.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-06-22T17:16:58.100 INFO:teuthology.orchestra.run.smithi101.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-06-22T17:16:58.100 INFO:teuthology.orchestra.run.smithi101.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-06-22T17:16:58.100 INFO:teuthology.orchestra.run.smithi101.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-06-22T17:16:58.100 INFO:teuthology.orchestra.run.smithi101.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-06-22T17:16:58.100 INFO:teuthology.orchestra.run.smithi101.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2023-06-22T17:16:58.100 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-06-22T17:16:58.100 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-06-22T17:16:58.101 INFO:teuthology.orchestra.run.smithi101.stdout:====================================================================================================================== 2023-06-22T17:16:58.101 INFO:teuthology.orchestra.run.smithi101.stdout:Install 118 Packages 2023-06-22T17:16:58.101 INFO:teuthology.orchestra.run.smithi101.stdout:Upgrade 2 Packages 2023-06-22T17:16:58.101 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-06-22T17:16:58.102 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 201 M 2023-06-22T17:16:58.102 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-06-22T17:16:58.114 INFO:teuthology.orchestra.run.smithi133.stdout:(6/120): ceph-mgr-17.2.6-719.g42ddc46e.el8.x86_ 6.5 MB/s | 1.6 MB 00:00 2023-06-22T17:16:58.608 INFO:teuthology.orchestra.run.smithi101.stdout:(1/120): ceph-17.2.6-719.g42ddc46e.el8.x86_64.r 36 kB/s | 6.4 kB 00:00 2023-06-22T17:16:58.698 INFO:teuthology.orchestra.run.smithi133.stdout:(7/120): ceph-common-17.2.6-719.g42ddc46e.el8.x 14 MB/s | 23 MB 00:01 2023-06-22T17:16:58.773 INFO:teuthology.orchestra.run.smithi133.stdout:(8/120): ceph-mon-17.2.6-719.g42ddc46e.el8.x86_ 5.6 MB/s | 3.9 MB 00:00 2023-06-22T17:16:58.859 INFO:teuthology.orchestra.run.smithi101.stdout:(2/120): ceph-fuse-17.2.6-719.g42ddc46e.el8.x86 3.4 MB/s | 873 kB 00:00 2023-06-22T17:16:58.931 INFO:teuthology.orchestra.run.smithi101.stdout:(3/120): ceph-immutable-object-cache-17.2.6-719 2.2 MB/s | 162 kB 00:00 2023-06-22T17:16:59.238 INFO:teuthology.orchestra.run.smithi133.stdout:(9/120): ceph-selinux-17.2.6-719.g42ddc46e.el8. 52 kB/s | 24 kB 00:00 2023-06-22T17:16:59.312 INFO:teuthology.orchestra.run.smithi101.stdout:(4/120): ceph-mds-17.2.6-719.g42ddc46e.el8.x86_ 6.0 MB/s | 2.3 MB 00:00 2023-06-22T17:16:59.535 INFO:teuthology.orchestra.run.smithi101.stdout:(5/120): ceph-mgr-17.2.6-719.g42ddc46e.el8.x86_ 7.0 MB/s | 1.6 MB 00:00 2023-06-22T17:16:59.571 INFO:teuthology.orchestra.run.smithi101.stdout:(6/120): ceph-base-17.2.6-719.g42ddc46e.el8.x86 4.5 MB/s | 5.2 MB 00:01 2023-06-22T17:17:00.008 INFO:teuthology.orchestra.run.smithi101.stdout:(7/120): ceph-mon-17.2.6-719.g42ddc46e.el8.x86_ 8.3 MB/s | 3.9 MB 00:00 2023-06-22T17:17:00.480 INFO:teuthology.orchestra.run.smithi039.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-22T17:17:00.481 INFO:teuthology.orchestra.run.smithi039.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-22T17:17:00.492 INFO:teuthology.orchestra.run.smithi039.stdout:Package NetworkManager-1:1.30.0-13.el8_4.x86_64 is already installed. 2023-06-22T17:17:00.493 INFO:teuthology.orchestra.run.smithi039.stdout:Package iproute-5.9.0-4.el8.x86_64 is already installed. 2023-06-22T17:17:00.494 INFO:teuthology.orchestra.run.smithi039.stdout:Package util-linux-2.32.1-27.el8.x86_64 is already installed. 2023-06-22T17:17:00.494 INFO:teuthology.orchestra.run.smithi039.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-06-22T17:17:00.496 INFO:teuthology.orchestra.run.smithi039.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-06-22T17:17:00.499 INFO:teuthology.orchestra.run.smithi039.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2023-06-22T17:17:00.500 INFO:teuthology.orchestra.run.smithi039.stdout:Package xfsprogs-5.0.0-8.el8.x86_64 is already installed. 2023-06-22T17:17:01.012 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-06-22T17:17:01.024 INFO:teuthology.orchestra.run.smithi039.stdout:====================================================================================================================== 2023-06-22T17:17:01.024 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-06-22T17:17:01.025 INFO:teuthology.orchestra.run.smithi039.stdout:====================================================================================================================== 2023-06-22T17:17:01.025 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-06-22T17:17:01.025 INFO:teuthology.orchestra.run.smithi039.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-06-22T17:17:01.025 INFO:teuthology.orchestra.run.smithi039.stdout: ceph x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 6.4 k 2023-06-22T17:17:01.025 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-base x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.2 M 2023-06-22T17:17:01.025 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 873 k 2023-06-22T17:17:01.025 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 162 k 2023-06-22T17:17:01.026 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 1.6 M 2023-06-22T17:17:01.026 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 121 k 2023-06-22T17:17:01.026 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-dashboard noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 1.7 M 2023-06-22T17:17:01.026 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 7.4 M 2023-06-22T17:17:01.026 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-rook noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 47 k 2023-06-22T17:17:01.026 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-radosgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-22T17:17:01.026 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-test x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 52 M 2023-06-22T17:17:01.027 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-volume noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 264 k 2023-06-22T17:17:01.027 INFO:teuthology.orchestra.run.smithi039.stdout: cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 80 k 2023-06-22T17:17:01.027 INFO:teuthology.orchestra.run.smithi039.stdout: cephfs-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 255 k 2023-06-22T17:17:01.027 INFO:teuthology.orchestra.run.smithi039.stdout: cephfs-top noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 18 k 2023-06-22T17:17:01.027 INFO:teuthology.orchestra.run.smithi039.stdout: elfutils-libelf-devel x86_64 0.182-3.el8 rhel-8-for-x86_64-baseos-rpms 59 k 2023-06-22T17:17:01.027 INFO:teuthology.orchestra.run.smithi039.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-06-22T17:17:01.027 INFO:teuthology.orchestra.run.smithi039.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-06-22T17:17:01.028 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 25 k 2023-06-22T17:17:01.028 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 794 k 2023-06-22T17:17:01.028 INFO:teuthology.orchestra.run.smithi039.stdout: librados-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 130 k 2023-06-22T17:17:01.028 INFO:teuthology.orchestra.run.smithi039.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-06-22T17:17:01.028 INFO:teuthology.orchestra.run.smithi039.stdout: libuuid-devel x86_64 2.32.1-27.el8 rhel-8-for-x86_64-baseos-rpms 97 k 2023-06-22T17:17:01.028 INFO:teuthology.orchestra.run.smithi039.stdout: openssl-devel x86_64 1:1.1.1g-15.el8_3 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-06-22T17:17:01.029 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cephfs x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 190 k 2023-06-22T17:17:01.029 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rados x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 371 k 2023-06-22T17:17:01.029 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 357 k 2023-06-22T17:17:01.029 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 107 k 2023-06-22T17:17:01.029 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 91 k 2023-06-22T17:17:01.029 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-22T17:17:01.029 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-nbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 173 k 2023-06-22T17:17:01.029 INFO:teuthology.orchestra.run.smithi039.stdout: xfsprogs-devel x86_64 5.0.0-8.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2023-06-22T17:17:01.030 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrading: 2023-06-22T17:17:01.030 INFO:teuthology.orchestra.run.smithi039.stdout: librados2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-22T17:17:01.030 INFO:teuthology.orchestra.run.smithi039.stdout: librbd1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.6 M 2023-06-22T17:17:01.030 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2023-06-22T17:17:01.030 INFO:teuthology.orchestra.run.smithi039.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-06-22T17:17:01.030 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 23 M 2023-06-22T17:17:01.030 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-grafana-dashboards noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 23 k 2023-06-22T17:17:01.030 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mds x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 2.3 M 2023-06-22T17:17:01.031 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-modules-core noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 238 k 2023-06-22T17:17:01.031 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mon x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.9 M 2023-06-22T17:17:01.031 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-osd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-22T17:17:01.031 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-prometheus-alerts noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 14 k 2023-06-22T17:17:01.031 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-selinux x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 24 k 2023-06-22T17:17:01.031 INFO:teuthology.orchestra.run.smithi039.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-06-22T17:17:01.032 INFO:teuthology.orchestra.run.smithi039.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-06-22T17:17:01.032 INFO:teuthology.orchestra.run.smithi039.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-06-22T17:17:01.032 INFO:teuthology.orchestra.run.smithi039.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-06-22T17:17:01.032 INFO:teuthology.orchestra.run.smithi039.stdout: krb5-devel x86_64 1.18.2-8.3.el8_4 rhel-8-for-x86_64-baseos-rpms 559 k 2023-06-22T17:17:01.032 INFO:teuthology.orchestra.run.smithi039.stdout: libcephsqlite x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 174 k 2023-06-22T17:17:01.032 INFO:teuthology.orchestra.run.smithi039.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2023-06-22T17:17:01.032 INFO:teuthology.orchestra.run.smithi039.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-06-22T17:17:01.033 INFO:teuthology.orchestra.run.smithi039.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-06-22T17:17:01.033 INFO:teuthology.orchestra.run.smithi039.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-06-22T17:17:01.033 INFO:teuthology.orchestra.run.smithi039.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-06-22T17:17:01.033 INFO:teuthology.orchestra.run.smithi039.stdout: libradosstriper1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 506 k 2023-06-22T17:17:01.033 INFO:teuthology.orchestra.run.smithi039.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-06-22T17:17:01.033 INFO:teuthology.orchestra.run.smithi039.stdout: librgw2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.5 M 2023-06-22T17:17:01.033 INFO:teuthology.orchestra.run.smithi039.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-06-22T17:17:01.033 INFO:teuthology.orchestra.run.smithi039.stdout: libsepol-devel x86_64 2.9-2.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2023-06-22T17:17:01.034 INFO:teuthology.orchestra.run.smithi039.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-06-22T17:17:01.034 INFO:teuthology.orchestra.run.smithi039.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-06-22T17:17:01.034 INFO:teuthology.orchestra.run.smithi039.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-06-22T17:17:01.034 INFO:teuthology.orchestra.run.smithi039.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-06-22T17:17:01.034 INFO:teuthology.orchestra.run.smithi039.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-06-22T17:17:01.034 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-06-22T17:17:01.034 INFO:teuthology.orchestra.run.smithi039.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-06-22T17:17:01.034 INFO:teuthology.orchestra.run.smithi039.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-06-22T17:17:01.035 INFO:teuthology.orchestra.run.smithi039.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-06-22T17:17:01.035 INFO:teuthology.orchestra.run.smithi039.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-06-22T17:17:01.035 INFO:teuthology.orchestra.run.smithi039.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-06-22T17:17:01.035 INFO:teuthology.orchestra.run.smithi039.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-06-22T17:17:01.035 INFO:teuthology.orchestra.run.smithi039.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-06-22T17:17:01.035 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-06-22T17:17:01.035 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-argparse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 45 k 2023-06-22T17:17:01.035 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 115 k 2023-06-22T17:17:01.036 INFO:teuthology.orchestra.run.smithi039.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-06-22T17:17:01.036 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-06-22T17:17:01.036 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-06-22T17:17:01.036 INFO:teuthology.orchestra.run.smithi039.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-06-22T17:17:01.036 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-06-22T17:17:01.036 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-06-22T17:17:01.036 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-06-22T17:17:01.037 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-06-22T17:17:01.037 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-06-22T17:17:01.037 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-22T17:17:01.037 INFO:teuthology.orchestra.run.smithi039.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-06-22T17:17:01.037 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-06-22T17:17:01.037 INFO:teuthology.orchestra.run.smithi039.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-06-22T17:17:01.037 INFO:teuthology.orchestra.run.smithi039.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-06-22T17:17:01.037 INFO:teuthology.orchestra.run.smithi039.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-06-22T17:17:01.038 INFO:teuthology.orchestra.run.smithi039.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-06-22T17:17:01.038 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-06-22T17:17:01.038 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-06-22T17:17:01.038 INFO:teuthology.orchestra.run.smithi039.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-06-22T17:17:01.038 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-06-22T17:17:01.038 INFO:teuthology.orchestra.run.smithi039.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-06-22T17:17:01.038 INFO:teuthology.orchestra.run.smithi039.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-06-22T17:17:01.038 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-06-22T17:17:01.039 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-06-22T17:17:01.039 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-06-22T17:17:01.039 INFO:teuthology.orchestra.run.smithi039.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-06-22T17:17:01.039 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-22T17:17:01.039 INFO:teuthology.orchestra.run.smithi039.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-06-22T17:17:01.039 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-06-22T17:17:01.039 INFO:teuthology.orchestra.run.smithi039.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-06-22T17:17:01.039 INFO:teuthology.orchestra.run.smithi039.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-06-22T17:17:01.040 INFO:teuthology.orchestra.run.smithi039.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-06-22T17:17:01.040 INFO:teuthology.orchestra.run.smithi039.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-06-22T17:17:01.040 INFO:teuthology.orchestra.run.smithi039.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-06-22T17:17:01.040 INFO:teuthology.orchestra.run.smithi039.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-06-22T17:17:01.040 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-06-22T17:17:01.040 INFO:teuthology.orchestra.run.smithi039.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-06-22T17:17:01.040 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-06-22T17:17:01.040 INFO:teuthology.orchestra.run.smithi039.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-06-22T17:17:01.041 INFO:teuthology.orchestra.run.smithi039.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-06-22T17:17:01.041 INFO:teuthology.orchestra.run.smithi039.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-06-22T17:17:01.041 INFO:teuthology.orchestra.run.smithi039.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-06-22T17:17:01.041 INFO:teuthology.orchestra.run.smithi039.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-06-22T17:17:01.041 INFO:teuthology.orchestra.run.smithi039.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2023-06-22T17:17:01.041 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:17:01.042 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-06-22T17:17:01.042 INFO:teuthology.orchestra.run.smithi039.stdout:====================================================================================================================== 2023-06-22T17:17:01.042 INFO:teuthology.orchestra.run.smithi039.stdout:Install 118 Packages 2023-06-22T17:17:01.042 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrade 2 Packages 2023-06-22T17:17:01.042 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:17:01.043 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 201 M 2023-06-22T17:17:01.043 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-06-22T17:17:01.464 INFO:teuthology.orchestra.run.smithi039.stdout:(1/120): ceph-17.2.6-719.g42ddc46e.el8.x86_64.r 23 kB/s | 6.4 kB 00:00 2023-06-22T17:17:01.983 INFO:teuthology.orchestra.run.smithi039.stdout:(2/120): ceph-fuse-17.2.6-719.g42ddc46e.el8.x86 1.6 MB/s | 873 kB 00:00 2023-06-22T17:17:02.008 INFO:teuthology.orchestra.run.smithi039.stdout:(3/120): ceph-immutable-object-cache-17.2.6-719 6.4 MB/s | 162 kB 00:00 2023-06-22T17:17:02.019 INFO:teuthology.orchestra.run.smithi133.stdout:(10/120): ceph-radosgw-17.2.6-719.g42ddc46e.el8 5.0 MB/s | 17 MB 00:03 2023-06-22T17:17:02.090 INFO:teuthology.orchestra.run.smithi101.stdout:(8/120): ceph-radosgw-17.2.6-719.g42ddc46e.el8. 8.0 MB/s | 17 MB 00:02 2023-06-22T17:17:02.124 INFO:teuthology.orchestra.run.smithi133.stdout:(11/120): ceph-osd-17.2.6-719.g42ddc46e.el8.x86 4.2 MB/s | 17 MB 00:04 2023-06-22T17:17:02.126 INFO:teuthology.orchestra.run.smithi133.stdout:(12/120): cephfs-mirror-17.2.6-719.g42ddc46e.el 2.3 MB/s | 255 kB 00:00 2023-06-22T17:17:02.140 INFO:teuthology.orchestra.run.smithi101.stdout:(9/120): ceph-selinux-17.2.6-719.g42ddc46e.el8. 486 kB/s | 24 kB 00:00 2023-06-22T17:17:02.187 INFO:teuthology.orchestra.run.smithi133.stdout:(13/120): libcephfs-devel-17.2.6-719.g42ddc46e. 399 kB/s | 25 kB 00:00 2023-06-22T17:17:02.389 INFO:teuthology.orchestra.run.smithi133.stdout:(14/120): libcephfs2-17.2.6-719.g42ddc46e.el8.x 3.0 MB/s | 794 kB 00:00 2023-06-22T17:17:02.449 INFO:teuthology.orchestra.run.smithi133.stdout:(15/120): librados-devel-17.2.6-719.g42ddc46e.e 2.1 MB/s | 130 kB 00:00 2023-06-22T17:17:02.498 INFO:teuthology.orchestra.run.smithi039.stdout:(4/120): ceph-mds-17.2.6-719.g42ddc46e.el8.x86_ 4.7 MB/s | 2.3 MB 00:00 2023-06-22T17:17:02.627 INFO:teuthology.orchestra.run.smithi133.stdout:(16/120): libradosstriper1-17.2.6-719.g42ddc46e 2.8 MB/s | 506 kB 00:00 2023-06-22T17:17:02.959 INFO:teuthology.orchestra.run.smithi039.stdout:(5/120): ceph-mgr-17.2.6-719.g42ddc46e.el8.x86_ 3.4 MB/s | 1.6 MB 00:00 2023-06-22T17:17:03.029 INFO:teuthology.orchestra.run.smithi101.stdout:(10/120): ceph-common-17.2.6-719.g42ddc46e.el8. 5.1 MB/s | 23 MB 00:04 2023-06-22T17:17:03.087 INFO:teuthology.orchestra.run.smithi133.stdout:(17/120): libcephsqlite-17.2.6-719.g42ddc46e.el 193 kB/s | 174 kB 00:00 2023-06-22T17:17:03.110 INFO:teuthology.orchestra.run.smithi101.stdout:(11/120): cephfs-mirror-17.2.6-719.g42ddc46e.el 3.1 MB/s | 255 kB 00:00 2023-06-22T17:17:03.167 INFO:teuthology.orchestra.run.smithi133.stdout:(18/120): python3-ceph-argparse-17.2.6-719.g42d 561 kB/s | 45 kB 00:00 2023-06-22T17:17:03.184 INFO:teuthology.orchestra.run.smithi101.stdout:(12/120): libcephfs-devel-17.2.6-719.g42ddc46e. 334 kB/s | 25 kB 00:00 2023-06-22T17:17:03.246 INFO:teuthology.orchestra.run.smithi133.stdout:(19/120): python3-ceph-common-17.2.6-719.g42ddc 1.4 MB/s | 115 kB 00:00 2023-06-22T17:17:03.378 INFO:teuthology.orchestra.run.smithi101.stdout:(13/120): libcephfs2-17.2.6-719.g42ddc46e.el8.x 4.0 MB/s | 794 kB 00:00 2023-06-22T17:17:03.470 INFO:teuthology.orchestra.run.smithi101.stdout:(14/120): libcephsqlite-17.2.6-719.g42ddc46e.el 1.9 MB/s | 174 kB 00:00 2023-06-22T17:17:03.541 INFO:teuthology.orchestra.run.smithi101.stdout:(15/120): librados-devel-17.2.6-719.g42ddc46e.e 1.8 MB/s | 130 kB 00:00 2023-06-22T17:17:03.687 INFO:teuthology.orchestra.run.smithi101.stdout:(16/120): libradosstriper1-17.2.6-719.g42ddc46e 3.4 MB/s | 506 kB 00:00 2023-06-22T17:17:03.766 INFO:teuthology.orchestra.run.smithi039.stdout:(6/120): ceph-base-17.2.6-719.g42ddc46e.el8.x86 2.0 MB/s | 5.2 MB 00:02 2023-06-22T17:17:03.975 INFO:teuthology.orchestra.run.smithi133.stdout:(20/120): python3-cephfs-17.2.6-719.g42ddc46e.e 261 kB/s | 190 kB 00:00 2023-06-22T17:17:03.981 INFO:teuthology.orchestra.run.smithi039.stdout:(7/120): ceph-mon-17.2.6-719.g42ddc46e.el8.x86_ 3.8 MB/s | 3.9 MB 00:01 2023-06-22T17:17:04.211 INFO:teuthology.orchestra.run.smithi133.stdout:(21/120): librgw2-17.2.6-719.g42ddc46e.el8.x86_ 3.5 MB/s | 5.5 MB 00:01 2023-06-22T17:17:04.239 INFO:teuthology.orchestra.run.smithi133.stdout:(22/120): python3-rados-17.2.6-719.g42ddc46e.el 1.4 MB/s | 371 kB 00:00 2023-06-22T17:17:04.316 INFO:teuthology.orchestra.run.smithi133.stdout:(23/120): python3-rgw-17.2.6-719.g42ddc46e.el8. 1.3 MB/s | 107 kB 00:00 2023-06-22T17:17:04.334 INFO:teuthology.orchestra.run.smithi133.stdout:(24/120): python3-rbd-17.2.6-719.g42ddc46e.el8. 2.8 MB/s | 357 kB 00:00 2023-06-22T17:17:04.390 INFO:teuthology.orchestra.run.smithi133.stdout:(25/120): rbd-fuse-17.2.6-719.g42ddc46e.el8.x86 1.2 MB/s | 91 kB 00:00 2023-06-22T17:17:04.541 INFO:teuthology.orchestra.run.smithi133.stdout:(26/120): rbd-nbd-17.2.6-719.g42ddc46e.el8.x86_ 1.1 MB/s | 173 kB 00:00 2023-06-22T17:17:04.576 INFO:teuthology.orchestra.run.smithi101.stdout:(17/120): ceph-osd-17.2.6-719.g42ddc46e.el8.x86 3.3 MB/s | 17 MB 00:05 2023-06-22T17:17:04.596 INFO:teuthology.orchestra.run.smithi133.stdout:(27/120): ceph-grafana-dashboards-17.2.6-719.g4 421 kB/s | 23 kB 00:00 2023-06-22T17:17:04.632 INFO:teuthology.orchestra.run.smithi101.stdout:(18/120): python3-ceph-argparse-17.2.6-719.g42d 794 kB/s | 45 kB 00:00 2023-06-22T17:17:04.673 INFO:teuthology.orchestra.run.smithi133.stdout:(28/120): ceph-mgr-cephadm-17.2.6-719.g42ddc46e 1.5 MB/s | 121 kB 00:00 2023-06-22T17:17:04.695 INFO:teuthology.orchestra.run.smithi101.stdout:(19/120): python3-ceph-common-17.2.6-719.g42ddc 1.8 MB/s | 115 kB 00:00 2023-06-22T17:17:04.827 INFO:teuthology.orchestra.run.smithi101.stdout:(20/120): python3-cephfs-17.2.6-719.g42ddc46e.e 1.4 MB/s | 190 kB 00:00 2023-06-22T17:17:04.872 INFO:teuthology.orchestra.run.smithi101.stdout:(21/120): librgw2-17.2.6-719.g42ddc46e.el8.x86_ 4.7 MB/s | 5.5 MB 00:01 2023-06-22T17:17:04.959 INFO:teuthology.orchestra.run.smithi101.stdout:(22/120): python3-rbd-17.2.6-719.g42ddc46e.el8. 4.0 MB/s | 357 kB 00:00 2023-06-22T17:17:05.008 INFO:teuthology.orchestra.run.smithi101.stdout:(23/120): python3-rados-17.2.6-719.g42ddc46e.el 2.0 MB/s | 371 kB 00:00 2023-06-22T17:17:05.032 INFO:teuthology.orchestra.run.smithi101.stdout:(24/120): python3-rgw-17.2.6-719.g42ddc46e.el8. 1.4 MB/s | 107 kB 00:00 2023-06-22T17:17:05.065 INFO:teuthology.orchestra.run.smithi101.stdout:(25/120): rbd-fuse-17.2.6-719.g42ddc46e.el8.x86 1.6 MB/s | 91 kB 00:00 2023-06-22T17:17:05.199 INFO:teuthology.orchestra.run.smithi101.stdout:(26/120): rbd-nbd-17.2.6-719.g42ddc46e.el8.x86_ 1.3 MB/s | 173 kB 00:00 2023-06-22T17:17:05.258 INFO:teuthology.orchestra.run.smithi101.stdout:(27/120): ceph-grafana-dashboards-17.2.6-719.g4 390 kB/s | 23 kB 00:00 2023-06-22T17:17:05.285 INFO:teuthology.orchestra.run.smithi133.stdout:(29/120): rbd-mirror-17.2.6-719.g42ddc46e.el8.x 3.6 MB/s | 3.5 MB 00:00 2023-06-22T17:17:05.319 INFO:teuthology.orchestra.run.smithi101.stdout:(28/120): ceph-mgr-cephadm-17.2.6-719.g42ddc46e 2.0 MB/s | 121 kB 00:00 2023-06-22T17:17:05.572 INFO:teuthology.orchestra.run.smithi133.stdout:(30/120): ceph-mgr-dashboard-17.2.6-719.g42ddc4 1.9 MB/s | 1.7 MB 00:00 2023-06-22T17:17:05.743 INFO:teuthology.orchestra.run.smithi133.stdout:(31/120): ceph-mgr-modules-core-17.2.6-719.g42d 1.4 MB/s | 238 kB 00:00 2023-06-22T17:17:05.825 INFO:teuthology.orchestra.run.smithi133.stdout:(32/120): ceph-mgr-rook-17.2.6-719.g42ddc46e.el 579 kB/s | 47 kB 00:00 2023-06-22T17:17:05.875 INFO:teuthology.orchestra.run.smithi101.stdout:(29/120): rbd-mirror-17.2.6-719.g42ddc46e.el8.x 4.1 MB/s | 3.5 MB 00:00 2023-06-22T17:17:05.898 INFO:teuthology.orchestra.run.smithi133.stdout:(33/120): ceph-prometheus-alerts-17.2.6-719.g42 196 kB/s | 14 kB 00:00 2023-06-22T17:17:06.080 INFO:teuthology.orchestra.run.smithi133.stdout:(34/120): ceph-volume-17.2.6-719.g42ddc46e.el8. 1.4 MB/s | 264 kB 00:00 2023-06-22T17:17:06.129 INFO:teuthology.orchestra.run.smithi101.stdout:(30/120): ceph-mgr-dashboard-17.2.6-719.g42ddc4 2.1 MB/s | 1.7 MB 00:00 2023-06-22T17:17:06.173 INFO:teuthology.orchestra.run.smithi133.stdout:(35/120): cephadm-17.2.6-719.g42ddc46e.el8.noar 859 kB/s | 80 kB 00:00 2023-06-22T17:17:06.250 INFO:teuthology.orchestra.run.smithi101.stdout:(31/120): ceph-mgr-modules-core-17.2.6-719.g42d 1.9 MB/s | 238 kB 00:00 2023-06-22T17:17:06.264 INFO:teuthology.orchestra.run.smithi133.stdout:(36/120): cephfs-top-17.2.6-719.g42ddc46e.el8.n 196 kB/s | 18 kB 00:00 2023-06-22T17:17:06.308 INFO:teuthology.orchestra.run.smithi101.stdout:(32/120): ceph-mgr-rook-17.2.6-719.g42ddc46e.el 824 kB/s | 47 kB 00:00 2023-06-22T17:17:06.362 INFO:teuthology.orchestra.run.smithi101.stdout:(33/120): ceph-prometheus-alerts-17.2.6-719.g42 268 kB/s | 14 kB 00:00 2023-06-22T17:17:06.482 INFO:teuthology.orchestra.run.smithi101.stdout:(34/120): ceph-volume-17.2.6-719.g42ddc46e.el8. 2.1 MB/s | 264 kB 00:00 2023-06-22T17:17:06.537 INFO:teuthology.orchestra.run.smithi101.stdout:(35/120): cephadm-17.2.6-719.g42ddc46e.el8.noar 1.4 MB/s | 80 kB 00:00 2023-06-22T17:17:06.548 INFO:teuthology.orchestra.run.smithi133.stdout:(37/120): gperftools-libs-2.7-9.el8.x86_64.rpm 1.1 MB/s | 306 kB 00:00 2023-06-22T17:17:06.596 INFO:teuthology.orchestra.run.smithi101.stdout:(36/120): cephfs-top-17.2.6-719.g42ddc46e.el8.n 304 kB/s | 18 kB 00:00 2023-06-22T17:17:06.603 INFO:teuthology.orchestra.run.smithi133.stdout:(38/120): liboath-2.6.2-3.el8.x86_64.rpm 1.1 MB/s | 59 kB 00:00 2023-06-22T17:17:06.644 INFO:teuthology.orchestra.run.smithi133.stdout:(39/120): libunwind-1.3.1-3.el8.x86_64.rpm 1.8 MB/s | 75 kB 00:00 2023-06-22T17:17:06.779 INFO:teuthology.orchestra.run.smithi133.stdout:(40/120): python3-asyncssh-2.7.0-2.el8.noarch.r 3.4 MB/s | 476 kB 00:00 2023-06-22T17:17:06.807 INFO:teuthology.orchestra.run.smithi133.stdout:(41/120): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 1.5 MB/s | 44 kB 00:00 2023-06-22T17:17:06.857 INFO:teuthology.orchestra.run.smithi133.stdout:(42/120): python3-beautifulsoup4-4.6.3-2.el8.1. 3.6 MB/s | 185 kB 00:00 2023-06-22T17:17:06.877 INFO:teuthology.orchestra.run.smithi133.stdout:(43/120): python3-cachetools-3.1.1-4.el8.noarch 1.7 MB/s | 33 kB 00:00 2023-06-22T17:17:06.896 INFO:teuthology.orchestra.run.smithi133.stdout:(44/120): python3-certifi-2018.10.15-7.el8.noar 856 kB/s | 16 kB 00:00 2023-06-22T17:17:06.936 INFO:teuthology.orchestra.run.smithi133.stdout:(45/120): python3-cheroot-8.5.2-1.el8.noarch.rp 4.3 MB/s | 173 kB 00:00 2023-06-22T17:17:07.004 INFO:teuthology.orchestra.run.smithi133.stdout:(46/120): python3-cherrypy-18.4.0-1.el8.noarch. 5.6 MB/s | 384 kB 00:00 2023-06-22T17:17:07.023 INFO:teuthology.orchestra.run.smithi133.stdout:(47/120): python3-google-auth-1.1.1-10.el8.noar 4.0 MB/s | 81 kB 00:00 2023-06-22T17:17:07.053 INFO:teuthology.orchestra.run.smithi133.stdout:(48/120): python3-jaraco-6.2-6.el8.noarch.rpm 386 kB/s | 11 kB 00:00 2023-06-22T17:17:07.076 INFO:teuthology.orchestra.run.smithi133.stdout:(49/120): python3-jaraco-functools-2.0-4.el8.no 749 kB/s | 18 kB 00:00 2023-06-22T17:17:07.079 INFO:teuthology.orchestra.run.smithi101.stdout:(37/120): gperftools-libs-2.7-9.el8.x86_64.rpm 634 kB/s | 306 kB 00:00 2023-06-22T17:17:07.170 INFO:teuthology.orchestra.run.smithi101.stdout:(38/120): liboath-2.6.2-3.el8.x86_64.rpm 651 kB/s | 59 kB 00:00 2023-06-22T17:17:07.208 INFO:teuthology.orchestra.run.smithi133.stdout:(50/120): ceph-mgr-diskprediction-local-17.2.6- 3.8 MB/s | 7.4 MB 00:01 2023-06-22T17:17:07.264 INFO:teuthology.orchestra.run.smithi101.stdout:(39/120): libunwind-1.3.1-3.el8.x86_64.rpm 797 kB/s | 75 kB 00:00 2023-06-22T17:17:07.274 INFO:teuthology.orchestra.run.smithi039.stdout:(8/120): ceph-common-17.2.6-719.g42ddc46e.el8.x 3.8 MB/s | 23 MB 00:06 2023-06-22T17:17:07.344 INFO:teuthology.orchestra.run.smithi133.stdout:(51/120): python3-kubernetes-11.0.0-6.el8.noarc 4.4 MB/s | 1.2 MB 00:00 2023-06-22T17:17:07.351 INFO:teuthology.orchestra.run.smithi039.stdout:(9/120): ceph-selinux-17.2.6-719.g42ddc46e.el8. 314 kB/s | 24 kB 00:00 2023-06-22T17:17:07.389 INFO:teuthology.orchestra.run.smithi133.stdout:(52/120): python3-logutils-0.3.5-11.el8.noarch. 269 kB/s | 49 kB 00:00 2023-06-22T17:17:07.393 INFO:teuthology.orchestra.run.smithi133.stdout:(53/120): python3-more-itertools-7.2.0-3.el8.no 1.2 MB/s | 59 kB 00:00 2023-06-22T17:17:07.473 INFO:teuthology.orchestra.run.smithi101.stdout:(40/120): python3-asyncssh-2.7.0-2.el8.noarch.r 2.2 MB/s | 476 kB 00:00 2023-06-22T17:17:07.486 INFO:teuthology.orchestra.run.smithi133.stdout:(54/120): python3-pecan-1.3.2-9.el8.noarch.rpm 3.0 MB/s | 283 kB 00:00 2023-06-22T17:17:07.489 INFO:teuthology.orchestra.run.smithi133.stdout:(55/120): python3-natsort-7.1.1-2.el8.noarch.rp 606 kB/s | 60 kB 00:00 2023-06-22T17:17:07.495 INFO:teuthology.orchestra.run.smithi101.stdout:(41/120): ceph-mgr-diskprediction-local-17.2.6- 4.6 MB/s | 7.4 MB 00:01 2023-06-22T17:17:07.526 INFO:teuthology.orchestra.run.smithi133.stdout:(56/120): python3-portend-2.6-1.el8.noarch.rpm 391 kB/s | 16 kB 00:00 2023-06-22T17:17:07.538 INFO:teuthology.orchestra.run.smithi133.stdout:(57/120): python3-repoze-lru-0.7-6.el8.noarch.r 689 kB/s | 34 kB 00:00 2023-06-22T17:17:07.571 INFO:teuthology.orchestra.run.smithi133.stdout:(58/120): python3-routes-2.4.1-12.el8.noarch.rp 4.2 MB/s | 196 kB 00:00 2023-06-22T17:17:07.573 INFO:teuthology.orchestra.run.smithi101.stdout:(42/120): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 432 kB/s | 44 kB 00:00 2023-06-22T17:17:07.587 INFO:teuthology.orchestra.run.smithi039.stdout:(10/120): ceph-radosgw-17.2.6-719.g42ddc46e.el8 4.6 MB/s | 17 MB 00:03 2023-06-22T17:17:07.610 INFO:teuthology.orchestra.run.smithi039.stdout:(11/120): cephfs-mirror-17.2.6-719.g42ddc46e.el 11 MB/s | 255 kB 00:00 2023-06-22T17:17:07.622 INFO:teuthology.orchestra.run.smithi133.stdout:(59/120): python3-simplegeneric-0.8.1-17.el8.no 371 kB/s | 19 kB 00:00 2023-06-22T17:17:07.639 INFO:teuthology.orchestra.run.smithi133.stdout:(60/120): python3-rsa-4.8-1.el8.noarch.rpm 611 kB/s | 61 kB 00:00 2023-06-22T17:17:07.642 INFO:teuthology.orchestra.run.smithi039.stdout:(12/120): libcephfs-devel-17.2.6-719.g42ddc46e. 797 kB/s | 25 kB 00:00 2023-06-22T17:17:07.653 INFO:teuthology.orchestra.run.smithi133.stdout:(61/120): python3-singledispatch-3.4.0.3-18.el8 757 kB/s | 24 kB 00:00 2023-06-22T17:17:07.656 INFO:teuthology.orchestra.run.smithi101.stdout:(43/120): python3-cachetools-3.1.1-4.el8.noarch 401 kB/s | 33 kB 00:00 2023-06-22T17:17:07.662 INFO:teuthology.orchestra.run.smithi133.stdout:(62/120): python3-tempora-1.14.1-5.el8.noarch.r 1.2 MB/s | 29 kB 00:00 2023-06-22T17:17:07.673 INFO:teuthology.orchestra.run.smithi133.stdout:(63/120): python3-trustme-0.6.0-4.el8.noarch.rp 1.4 MB/s | 27 kB 00:00 2023-06-22T17:17:07.719 INFO:teuthology.orchestra.run.smithi133.stdout:(64/120): python3-webob-1.8.5-1.el8.1.noarch.rp 5.4 MB/s | 251 kB 00:00 2023-06-22T17:17:07.735 INFO:teuthology.orchestra.run.smithi101.stdout:(44/120): python3-certifi-2018.10.15-7.el8.noar 205 kB/s | 16 kB 00:00 2023-06-22T17:17:07.745 INFO:teuthology.orchestra.run.smithi133.stdout:(65/120): python3-websocket-client-0.56.0-5.el8 2.3 MB/s | 61 kB 00:00 2023-06-22T17:17:07.752 INFO:teuthology.orchestra.run.smithi039.stdout:(13/120): libcephfs2-17.2.6-719.g42ddc46e.el8.x 7.0 MB/s | 794 kB 00:00 2023-06-22T17:17:07.765 INFO:teuthology.orchestra.run.smithi133.stdout:(66/120): python3-webtest-2.0.33-1.el8.noarch.r 4.2 MB/s | 85 kB 00:00 2023-06-22T17:17:07.785 INFO:teuthology.orchestra.run.smithi133.stdout:(67/120): python3-zc-lockfile-2.0-2.el8.noarch. 1.1 MB/s | 23 kB 00:00 2023-06-22T17:17:07.793 INFO:teuthology.orchestra.run.smithi039.stdout:(14/120): libcephsqlite-17.2.6-719.g42ddc46e.el 4.2 MB/s | 174 kB 00:00 2023-06-22T17:17:07.799 INFO:teuthology.orchestra.run.smithi133.stdout:(68/120): python3-waitress-1.4.3-1.el8.noarch.r 1.8 MB/s | 255 kB 00:00 2023-06-22T17:17:07.808 INFO:teuthology.orchestra.run.smithi133.stdout:(69/120): xmlstarlet-1.6.1-20.el8.x86_64.rpm 7.8 MB/s | 70 kB 00:00 2023-06-22T17:17:07.820 INFO:teuthology.orchestra.run.smithi101.stdout:(45/120): python3-cheroot-8.5.2-1.el8.noarch.rp 2.0 MB/s | 173 kB 00:00 2023-06-22T17:17:07.830 INFO:teuthology.orchestra.run.smithi039.stdout:(15/120): librados-devel-17.2.6-719.g42ddc46e.e 3.4 MB/s | 130 kB 00:00 2023-06-22T17:17:07.835 INFO:teuthology.orchestra.run.smithi133.stdout:(70/120): libaio-devel-0.3.112-1.el8.x86_64.rpm 704 kB/s | 19 kB 00:00 2023-06-22T17:17:07.847 INFO:teuthology.orchestra.run.smithi133.stdout:(71/120): python3-jwt-1.6.1-2.el8.noarch.rpm 3.5 MB/s | 43 kB 00:00 2023-06-22T17:17:07.860 INFO:teuthology.orchestra.run.smithi133.stdout:(72/120): python3-oauthlib-2.1.0-1.el8.noarch.r 12 MB/s | 155 kB 00:00 2023-06-22T17:17:07.874 INFO:teuthology.orchestra.run.smithi133.stdout:(73/120): pcre2-utf32-10.32-2.el8.x86_64.rpm 16 MB/s | 220 kB 00:00 2023-06-22T17:17:07.888 INFO:teuthology.orchestra.run.smithi133.stdout:(74/120): keyutils-libs-devel-1.5.10-6.el8.x86_ 3.2 MB/s | 48 kB 00:00 2023-06-22T17:17:07.898 INFO:teuthology.orchestra.run.smithi039.stdout:(16/120): libradosstriper1-17.2.6-719.g42ddc46e 7.3 MB/s | 506 kB 00:00 2023-06-22T17:17:07.906 INFO:teuthology.orchestra.run.smithi133.stdout:(75/120): pcre2-utf16-10.32-2.el8.x86_64.rpm 13 MB/s | 229 kB 00:00 2023-06-22T17:17:07.920 INFO:teuthology.orchestra.run.smithi133.stdout:(76/120): python3-requests-oauthlib-1.0.0-1.el8 2.9 MB/s | 43 kB 00:00 2023-06-22T17:17:07.921 INFO:teuthology.orchestra.run.smithi101.stdout:(46/120): python3-beautifulsoup4-4.6.3-2.el8.1. 435 kB/s | 185 kB 00:00 2023-06-22T17:17:07.960 INFO:teuthology.orchestra.run.smithi133.stdout:(77/120): openssl-devel-1.1.1g-15.el8_3.x86_64. 59 MB/s | 2.3 MB 00:00 2023-06-22T17:17:07.974 INFO:teuthology.orchestra.run.smithi133.stdout:(78/120): libverto-devel-0.3.0-5.el8.x86_64.rpm 1.2 MB/s | 18 kB 00:00 2023-06-22T17:17:07.997 INFO:teuthology.orchestra.run.smithi133.stdout:(79/120): pcre2-devel-10.32-2.el8.x86_64.rpm 26 MB/s | 605 kB 00:00 2023-06-22T17:17:08.004 INFO:teuthology.orchestra.run.smithi101.stdout:(47/120): python3-google-auth-1.1.1-10.el8.noar 993 kB/s | 81 kB 00:00 2023-06-22T17:17:08.006 INFO:teuthology.orchestra.run.smithi101.stdout:(48/120): python3-cherrypy-18.4.0-1.el8.noarch. 2.0 MB/s | 384 kB 00:00 2023-06-22T17:17:08.013 INFO:teuthology.orchestra.run.smithi133.stdout:(80/120): fuse-2.9.7-12.el8.x86_64.rpm 5.0 MB/s | 83 kB 00:00 2023-06-22T17:17:08.026 INFO:teuthology.orchestra.run.smithi133.stdout:(81/120): thrift-0.13.0-2.el8.x86_64.rpm 7.2 MB/s | 1.7 MB 00:00 2023-06-22T17:17:08.028 INFO:teuthology.orchestra.run.smithi133.stdout:(82/120): elfutils-libelf-devel-0.182-3.el8.x86 3.9 MB/s | 59 kB 00:00 2023-06-22T17:17:08.041 INFO:teuthology.orchestra.run.smithi133.stdout:(83/120): libcom_err-devel-1.45.6-1.el8.x86_64. 3.0 MB/s | 38 kB 00:00 2023-06-22T17:17:08.055 INFO:teuthology.orchestra.run.smithi133.stdout:(84/120): libquadmath-8.4.1-1.el8.x86_64.rpm 5.7 MB/s | 169 kB 00:00 2023-06-22T17:17:08.056 INFO:teuthology.orchestra.run.smithi133.stdout:(85/120): libsepol-devel-2.9-2.el8.x86_64.rpm 5.6 MB/s | 86 kB 00:00 2023-06-22T17:17:08.067 INFO:teuthology.orchestra.run.smithi133.stdout:(86/120): zlib-devel-1.2.11-17.el8.x86_64.rpm 4.5 MB/s | 58 kB 00:00 2023-06-22T17:17:08.070 INFO:teuthology.orchestra.run.smithi133.stdout:(87/120): librabbitmq-0.9.0-3.el8.x86_64.rpm 3.3 MB/s | 47 kB 00:00 2023-06-22T17:17:08.074 INFO:teuthology.orchestra.run.smithi101.stdout:(49/120): python3-jaraco-6.2-6.el8.noarch.rpm 157 kB/s | 11 kB 00:00 2023-06-22T17:17:08.084 INFO:teuthology.orchestra.run.smithi101.stdout:(50/120): python3-jaraco-functools-2.0-4.el8.no 224 kB/s | 18 kB 00:00 2023-06-22T17:17:08.086 INFO:teuthology.orchestra.run.smithi133.stdout:(88/120): krb5-devel-1.18.2-8.3.el8_4.x86_64.rp 30 MB/s | 559 kB 00:00 2023-06-22T17:17:08.092 INFO:teuthology.orchestra.run.smithi133.stdout:(89/120): libgfortran-8.4.1-1.el8.x86_64.rpm 29 MB/s | 639 kB 00:00 2023-06-22T17:17:08.099 INFO:teuthology.orchestra.run.smithi133.stdout:(90/120): xfsprogs-devel-5.0.0-8.el8.x86_64.rpm 5.2 MB/s | 70 kB 00:00 2023-06-22T17:17:08.110 INFO:teuthology.orchestra.run.smithi133.stdout:(91/120): libselinux-devel-2.9-5.el8.x86_64.rpm 11 MB/s | 200 kB 00:00 2023-06-22T17:17:08.118 INFO:teuthology.orchestra.run.smithi133.stdout:(92/120): libuuid-devel-2.32.1-27.el8.x86_64.rp 5.2 MB/s | 97 kB 00:00 2023-06-22T17:17:08.146 INFO:teuthology.orchestra.run.smithi133.stdout:(93/120): python3-prettytable-0.7.2-14.el8.noar 1.5 MB/s | 44 kB 00:00 2023-06-22T17:17:08.159 INFO:teuthology.orchestra.run.smithi101.stdout:(51/120): python3-logutils-0.3.5-11.el8.noarch. 658 kB/s | 49 kB 00:00 2023-06-22T17:17:08.171 INFO:teuthology.orchestra.run.smithi133.stdout:(94/120): python3-jsonpatch-1.21-2.el8.noarch.r 1.1 MB/s | 27 kB 00:00 2023-06-22T17:17:08.202 INFO:teuthology.orchestra.run.smithi133.stdout:(95/120): jq-1.5-12.el8.x86_64.rpm 5.1 MB/s | 161 kB 00:00 2023-06-22T17:17:08.230 INFO:teuthology.orchestra.run.smithi133.stdout:(96/120): flex-2.6.1-9.el8.x86_64.rpm 11 MB/s | 320 kB 00:00 2023-06-22T17:17:08.234 INFO:teuthology.orchestra.run.smithi101.stdout:(52/120): python3-more-itertools-7.2.0-3.el8.no 782 kB/s | 59 kB 00:00 2023-06-22T17:17:08.308 INFO:teuthology.orchestra.run.smithi101.stdout:(53/120): python3-natsort-7.1.1-2.el8.noarch.rp 813 kB/s | 60 kB 00:00 2023-06-22T17:17:08.362 INFO:teuthology.orchestra.run.smithi133.stdout:(97/120): python3-scipy-1.0.0-20.module+el8.1.0 58 MB/s | 15 MB 00:00 2023-06-22T17:17:08.365 INFO:teuthology.orchestra.run.smithi133.stdout:(98/120): python3-werkzeug-0.12.2-4.el8.noarch. 3.3 MB/s | 457 kB 00:00 2023-06-22T17:17:08.390 INFO:teuthology.orchestra.run.smithi101.stdout:(54/120): python3-pecan-1.3.2-9.el8.noarch.rpm 3.4 MB/s | 283 kB 00:00 2023-06-22T17:17:08.395 INFO:teuthology.orchestra.run.smithi133.stdout:(99/120): python3-markupsafe-0.23-19.el8.x86_64 1.3 MB/s | 39 kB 00:00 2023-06-22T17:17:08.429 INFO:teuthology.orchestra.run.smithi133.stdout:(100/120): bison-3.0.4-10.el8.x86_64.rpm 20 MB/s | 688 kB 00:00 2023-06-22T17:17:08.447 INFO:teuthology.orchestra.run.smithi133.stdout:(101/120): python3-babel-2.5.1-5.el8.noarch.rpm 57 MB/s | 4.8 MB 00:00 2023-06-22T17:17:08.457 INFO:teuthology.orchestra.run.smithi133.stdout:(102/120): lttng-ust-2.8.1-11.el8.x86_64.rpm 9.1 MB/s | 259 kB 00:00 2023-06-22T17:17:08.464 INFO:teuthology.orchestra.run.smithi101.stdout:(55/120): python3-portend-2.6-1.el8.noarch.rpm 212 kB/s | 16 kB 00:00 2023-06-22T17:17:08.482 INFO:teuthology.orchestra.run.smithi133.stdout:(103/120): python3-numpy-f2py-1.14.3-9.el8.x86_ 6.3 MB/s | 225 kB 00:00 2023-06-22T17:17:08.487 INFO:teuthology.orchestra.run.smithi133.stdout:(104/120): python3-pyasn1-0.3.7-6.el8.noarch.rp 4.2 MB/s | 126 kB 00:00 2023-06-22T17:17:08.505 INFO:teuthology.orchestra.run.smithi101.stdout:(56/120): python3-kubernetes-11.0.0-6.el8.noarc 2.7 MB/s | 1.2 MB 00:00 2023-06-22T17:17:08.521 INFO:teuthology.orchestra.run.smithi133.stdout:(105/120): oniguruma-6.8.2-2.el8.x86_64.rpm 5.4 MB/s | 187 kB 00:00 2023-06-22T17:17:08.539 INFO:teuthology.orchestra.run.smithi101.stdout:(57/120): python3-repoze-lru-0.7-6.el8.noarch.r 453 kB/s | 34 kB 00:00 2023-06-22T17:17:08.555 INFO:teuthology.orchestra.run.smithi133.stdout:(106/120): python3-numpy-1.14.3-9.el8.x86_64.rp 51 MB/s | 3.7 MB 00:00 2023-06-22T17:17:08.561 INFO:teuthology.orchestra.run.smithi133.stdout:(107/120): python3-jinja2-2.10.1-2.el8_0.noarch 13 MB/s | 538 kB 00:00 2023-06-22T17:17:08.583 INFO:teuthology.orchestra.run.smithi101.stdout:(58/120): python3-routes-2.4.1-12.el8.noarch.rp 2.4 MB/s | 196 kB 00:00 2023-06-22T17:17:08.587 INFO:teuthology.orchestra.run.smithi133.stdout:(108/120): socat-1.7.3.3-2.el8.x86_64.rpm 9.4 MB/s | 302 kB 00:00 2023-06-22T17:17:08.595 INFO:teuthology.orchestra.run.smithi133.stdout:(109/120): python3-mako-1.0.6-13.el8.noarch.rpm 4.4 MB/s | 157 kB 00:00 2023-06-22T17:17:08.614 INFO:teuthology.orchestra.run.smithi101.stdout:(59/120): python3-rsa-4.8-1.el8.noarch.rpm 821 kB/s | 61 kB 00:00 2023-06-22T17:17:08.617 INFO:teuthology.orchestra.run.smithi133.stdout:(110/120): python3-pyasn1-modules-0.3.7-6.el8.n 3.6 MB/s | 110 kB 00:00 2023-06-22T17:17:08.633 INFO:teuthology.orchestra.run.smithi133.stdout:(111/120): librdkafka-0.11.4-1.el8.x86_64.rpm 9.3 MB/s | 353 kB 00:00 2023-06-22T17:17:08.648 INFO:teuthology.orchestra.run.smithi133.stdout:(112/120): python3-jsonpointer-1.10-11.el8.noar 655 kB/s | 20 kB 00:00 2023-06-22T17:17:08.654 INFO:teuthology.orchestra.run.smithi101.stdout:(60/120): python3-simplegeneric-0.8.1-17.el8.no 265 kB/s | 19 kB 00:00 2023-06-22T17:17:08.671 INFO:teuthology.orchestra.run.smithi133.stdout:(113/120): libtool-2.4.6-25.el8.x86_64.rpm 18 MB/s | 709 kB 00:00 2023-06-22T17:17:08.675 INFO:teuthology.orchestra.run.smithi133.stdout:(114/120): python3-pytz-2017.2-9.el8.noarch.rpm 1.9 MB/s | 54 kB 00:00 2023-06-22T17:17:08.688 INFO:teuthology.orchestra.run.smithi101.stdout:(61/120): python3-singledispatch-3.4.0.3-18.el8 324 kB/s | 24 kB 00:00 2023-06-22T17:17:08.708 INFO:teuthology.orchestra.run.smithi039.stdout:(17/120): librgw2-17.2.6-719.g42ddc46e.el8.x86_ 6.8 MB/s | 5.5 MB 00:00 2023-06-22T17:17:08.737 INFO:teuthology.orchestra.run.smithi039.stdout:(18/120): python3-ceph-argparse-17.2.6-719.g42d 1.5 MB/s | 45 kB 00:00 2023-06-22T17:17:08.748 INFO:teuthology.orchestra.run.smithi101.stdout:(62/120): python3-tempora-1.14.1-5.el8.noarch.r 312 kB/s | 29 kB 00:00 2023-06-22T17:17:08.759 INFO:teuthology.orchestra.run.smithi133.stdout:(115/120): openblas-threads-0.3.12-1.el8.x86_64 54 MB/s | 4.7 MB 00:00 2023-06-22T17:17:08.762 INFO:teuthology.orchestra.run.smithi039.stdout:(19/120): python3-ceph-common-17.2.6-719.g42ddc 4.5 MB/s | 115 kB 00:00 2023-06-22T17:17:08.774 INFO:teuthology.orchestra.run.smithi133.stdout:(116/120): openblas-0.3.12-1.el8.x86_64.rpm 46 MB/s | 4.6 MB 00:00 2023-06-22T17:17:08.782 INFO:teuthology.orchestra.run.smithi101.stdout:(63/120): python3-trustme-0.6.0-4.el8.noarch.rp 292 kB/s | 27 kB 00:00 2023-06-22T17:17:08.791 INFO:teuthology.orchestra.run.smithi133.stdout:(117/120): automake-1.16.1-7.el8.noarch.rpm 22 MB/s | 713 kB 00:00 2023-06-22T17:17:08.796 INFO:teuthology.orchestra.run.smithi039.stdout:(20/120): python3-cephfs-17.2.6-719.g42ddc46e.e 5.4 MB/s | 190 kB 00:00 2023-06-22T17:17:08.859 INFO:teuthology.orchestra.run.smithi039.stdout:(21/120): python3-rados-17.2.6-719.g42ddc46e.el 5.8 MB/s | 371 kB 00:00 2023-06-22T17:17:08.860 INFO:teuthology.orchestra.run.smithi101.stdout:(64/120): python3-webob-1.8.5-1.el8.1.noarch.rp 3.1 MB/s | 251 kB 00:00 2023-06-22T17:17:08.906 INFO:teuthology.orchestra.run.smithi101.stdout:(65/120): python3-waitress-1.4.3-1.el8.noarch.r 1.6 MB/s | 255 kB 00:00 2023-06-22T17:17:08.935 INFO:teuthology.orchestra.run.smithi101.stdout:(66/120): python3-websocket-client-0.56.0-5.el8 825 kB/s | 61 kB 00:00 2023-06-22T17:17:08.949 INFO:teuthology.orchestra.run.smithi039.stdout:(22/120): python3-rbd-17.2.6-719.g42ddc46e.el8. 3.9 MB/s | 357 kB 00:00 2023-06-22T17:17:08.978 INFO:teuthology.orchestra.run.smithi101.stdout:(67/120): python3-webtest-2.0.33-1.el8.noarch.r 1.2 MB/s | 85 kB 00:00 2023-06-22T17:17:08.999 INFO:teuthology.orchestra.run.smithi039.stdout:(23/120): python3-rgw-17.2.6-719.g42ddc46e.el8. 2.1 MB/s | 107 kB 00:00 2023-06-22T17:17:09.009 INFO:teuthology.orchestra.run.smithi101.stdout:(68/120): python3-zc-lockfile-2.0-2.el8.noarch. 311 kB/s | 23 kB 00:00 2023-06-22T17:17:09.013 INFO:teuthology.orchestra.run.smithi101.stdout:(69/120): xmlstarlet-1.6.1-20.el8.x86_64.rpm 19 MB/s | 70 kB 00:00 2023-06-22T17:17:09.047 INFO:teuthology.orchestra.run.smithi039.stdout:(24/120): rbd-fuse-17.2.6-719.g42ddc46e.el8.x86 1.8 MB/s | 91 kB 00:00 2023-06-22T17:17:09.279 INFO:teuthology.orchestra.run.smithi101.stdout:(70/120): python3-scipy-1.0.0-20.module+el8.1.0 55 MB/s | 15 MB 00:00 2023-06-22T17:17:09.293 INFO:teuthology.orchestra.run.smithi101.stdout:(71/120): python3-prettytable-0.7.2-14.el8.noar 3.0 MB/s | 44 kB 00:00 2023-06-22T17:17:09.306 INFO:teuthology.orchestra.run.smithi101.stdout:(72/120): python3-jsonpatch-1.21-2.el8.noarch.r 2.1 MB/s | 27 kB 00:00 2023-06-22T17:17:09.321 INFO:teuthology.orchestra.run.smithi101.stdout:(73/120): jq-1.5-12.el8.x86_64.rpm 11 MB/s | 161 kB 00:00 2023-06-22T17:17:09.337 INFO:teuthology.orchestra.run.smithi101.stdout:(74/120): flex-2.6.1-9.el8.x86_64.rpm 20 MB/s | 320 kB 00:00 2023-06-22T17:17:09.354 INFO:teuthology.orchestra.run.smithi101.stdout:(75/120): python3-werkzeug-0.12.2-4.el8.noarch. 26 MB/s | 457 kB 00:00 2023-06-22T17:17:09.421 INFO:teuthology.orchestra.run.smithi101.stdout:(76/120): python3-babel-2.5.1-5.el8.noarch.rpm 72 MB/s | 4.8 MB 00:00 2023-06-22T17:17:09.436 INFO:teuthology.orchestra.run.smithi101.stdout:(77/120): python3-markupsafe-0.23-19.el8.x86_64 2.5 MB/s | 39 kB 00:00 2023-06-22T17:17:09.458 INFO:teuthology.orchestra.run.smithi101.stdout:(78/120): bison-3.0.4-10.el8.x86_64.rpm 32 MB/s | 688 kB 00:00 2023-06-22T17:17:09.473 INFO:teuthology.orchestra.run.smithi101.stdout:(79/120): lttng-ust-2.8.1-11.el8.x86_64.rpm 17 MB/s | 259 kB 00:00 2023-06-22T17:17:09.488 INFO:teuthology.orchestra.run.smithi101.stdout:(80/120): python3-numpy-f2py-1.14.3-9.el8.x86_6 15 MB/s | 225 kB 00:00 2023-06-22T17:17:09.502 INFO:teuthology.orchestra.run.smithi101.stdout:(81/120): python3-pyasn1-0.3.7-6.el8.noarch.rpm 9.1 MB/s | 126 kB 00:00 2023-06-22T17:17:09.527 INFO:teuthology.orchestra.run.smithi101.stdout:(82/120): thrift-0.13.0-2.el8.x86_64.rpm 3.2 MB/s | 1.7 MB 00:00 2023-06-22T17:17:09.556 INFO:teuthology.orchestra.run.smithi101.stdout:(83/120): oniguruma-6.8.2-2.el8.x86_64.rpm 6.4 MB/s | 187 kB 00:00 2023-06-22T17:17:09.567 INFO:teuthology.orchestra.run.smithi101.stdout:(84/120): python3-numpy-1.14.3-9.el8.x86_64.rpm 58 MB/s | 3.7 MB 00:00 2023-06-22T17:17:09.574 INFO:teuthology.orchestra.run.smithi101.stdout:(85/120): python3-jinja2-2.10.1-2.el8_0.noarch. 28 MB/s | 538 kB 00:00 2023-06-22T17:17:09.584 INFO:teuthology.orchestra.run.smithi101.stdout:(86/120): socat-1.7.3.3-2.el8.x86_64.rpm 18 MB/s | 302 kB 00:00 2023-06-22T17:17:09.590 INFO:teuthology.orchestra.run.smithi101.stdout:(87/120): python3-mako-1.0.6-13.el8.noarch.rpm 10 MB/s | 157 kB 00:00 2023-06-22T17:17:09.599 INFO:teuthology.orchestra.run.smithi101.stdout:(88/120): python3-pyasn1-modules-0.3.7-6.el8.no 7.4 MB/s | 110 kB 00:00 2023-06-22T17:17:09.607 INFO:teuthology.orchestra.run.smithi101.stdout:(89/120): librdkafka-0.11.4-1.el8.x86_64.rpm 21 MB/s | 353 kB 00:00 2023-06-22T17:17:09.612 INFO:teuthology.orchestra.run.smithi101.stdout:(90/120): python3-jsonpointer-1.10-11.el8.noarc 1.5 MB/s | 20 kB 00:00 2023-06-22T17:17:09.626 INFO:teuthology.orchestra.run.smithi101.stdout:(91/120): python3-pytz-2017.2-9.el8.noarch.rpm 3.9 MB/s | 54 kB 00:00 2023-06-22T17:17:09.632 INFO:teuthology.orchestra.run.smithi101.stdout:(92/120): libtool-2.4.6-25.el8.x86_64.rpm 28 MB/s | 709 kB 00:00 2023-06-22T17:17:09.700 INFO:teuthology.orchestra.run.smithi101.stdout:(93/120): openblas-threads-0.3.12-1.el8.x86_64. 64 MB/s | 4.7 MB 00:00 2023-06-22T17:17:09.719 INFO:teuthology.orchestra.run.smithi101.stdout:(94/120): openblas-0.3.12-1.el8.x86_64.rpm 53 MB/s | 4.6 MB 00:00 2023-06-22T17:17:09.727 INFO:teuthology.orchestra.run.smithi101.stdout:(95/120): automake-1.16.1-7.el8.noarch.rpm 27 MB/s | 713 kB 00:00 2023-06-22T17:17:09.732 INFO:teuthology.orchestra.run.smithi101.stdout:(96/120): libaio-devel-0.3.112-1.el8.x86_64.rpm 1.5 MB/s | 19 kB 00:00 2023-06-22T17:17:09.740 INFO:teuthology.orchestra.run.smithi133.stdout:(118/120): librbd1-17.2.6-719.g42ddc46e.el8.x86 3.8 MB/s | 3.6 MB 00:00 2023-06-22T17:17:09.741 INFO:teuthology.orchestra.run.smithi101.stdout:(97/120): python3-jwt-1.6.1-2.el8.noarch.rpm 3.0 MB/s | 43 kB 00:00 2023-06-22T17:17:09.747 INFO:teuthology.orchestra.run.smithi101.stdout:(98/120): python3-oauthlib-2.1.0-1.el8.noarch.r 10 MB/s | 155 kB 00:00 2023-06-22T17:17:09.757 INFO:teuthology.orchestra.run.smithi101.stdout:(99/120): pcre2-utf32-10.32-2.el8.x86_64.rpm 13 MB/s | 220 kB 00:00 2023-06-22T17:17:09.761 INFO:teuthology.orchestra.run.smithi101.stdout:(100/120): keyutils-libs-devel-1.5.10-6.el8.x86 3.5 MB/s | 48 kB 00:00 2023-06-22T17:17:09.773 INFO:teuthology.orchestra.run.smithi039.stdout:(25/120): rbd-mirror-17.2.6-719.g42ddc46e.el8.x 4.8 MB/s | 3.5 MB 00:00 2023-06-22T17:17:09.774 INFO:teuthology.orchestra.run.smithi101.stdout:(101/120): pcre2-utf16-10.32-2.el8.x86_64.rpm 14 MB/s | 229 kB 00:00 2023-06-22T17:17:09.774 INFO:teuthology.orchestra.run.smithi101.stdout:(102/120): python3-requests-oauthlib-1.0.0-1.el 3.0 MB/s | 43 kB 00:00 2023-06-22T17:17:09.788 INFO:teuthology.orchestra.run.smithi101.stdout:(103/120): libverto-devel-0.3.0-5.el8.x86_64.rp 1.3 MB/s | 18 kB 00:00 2023-06-22T17:17:09.809 INFO:teuthology.orchestra.run.smithi101.stdout:(104/120): pcre2-devel-10.32-2.el8.x86_64.rpm 29 MB/s | 605 kB 00:00 2023-06-22T17:17:09.817 INFO:teuthology.orchestra.run.smithi101.stdout:(105/120): openssl-devel-1.1.1g-15.el8_3.x86_64 53 MB/s | 2.3 MB 00:00 2023-06-22T17:17:09.820 INFO:teuthology.orchestra.run.smithi039.stdout:(26/120): rbd-nbd-17.2.6-719.g42ddc46e.el8.x86_ 3.7 MB/s | 173 kB 00:00 2023-06-22T17:17:09.822 INFO:teuthology.orchestra.run.smithi101.stdout:(106/120): fuse-2.9.7-12.el8.x86_64.rpm 6.1 MB/s | 83 kB 00:00 2023-06-22T17:17:09.831 INFO:teuthology.orchestra.run.smithi101.stdout:(107/120): elfutils-libelf-devel-0.182-3.el8.x8 4.1 MB/s | 59 kB 00:00 2023-06-22T17:17:09.838 INFO:teuthology.orchestra.run.smithi101.stdout:(108/120): libquadmath-8.4.1-1.el8.x86_64.rpm 11 MB/s | 169 kB 00:00 2023-06-22T17:17:09.846 INFO:teuthology.orchestra.run.smithi101.stdout:(109/120): libcom_err-devel-1.45.6-1.el8.x86_64 2.6 MB/s | 38 kB 00:00 2023-06-22T17:17:09.853 INFO:teuthology.orchestra.run.smithi101.stdout:(110/120): libsepol-devel-2.9-2.el8.x86_64.rpm 5.8 MB/s | 86 kB 00:00 2023-06-22T17:17:09.861 INFO:teuthology.orchestra.run.smithi101.stdout:(111/120): zlib-devel-1.2.11-17.el8.x86_64.rpm 3.9 MB/s | 58 kB 00:00 2023-06-22T17:17:09.864 INFO:teuthology.orchestra.run.smithi039.stdout:(27/120): ceph-grafana-dashboards-17.2.6-719.g4 516 kB/s | 23 kB 00:00 2023-06-22T17:17:09.868 INFO:teuthology.orchestra.run.smithi101.stdout:(112/120): librabbitmq-0.9.0-3.el8.x86_64.rpm 3.1 MB/s | 47 kB 00:00 2023-06-22T17:17:09.881 INFO:teuthology.orchestra.run.smithi101.stdout:(113/120): krb5-devel-1.18.2-8.3.el8_4.x86_64.r 27 MB/s | 559 kB 00:00 2023-06-22T17:17:09.888 INFO:teuthology.orchestra.run.smithi101.stdout:(114/120): libgfortran-8.4.1-1.el8.x86_64.rpm 32 MB/s | 639 kB 00:00 2023-06-22T17:17:09.896 INFO:teuthology.orchestra.run.smithi101.stdout:(115/120): xfsprogs-devel-5.0.0-8.el8.x86_64.rp 4.7 MB/s | 70 kB 00:00 2023-06-22T17:17:09.906 INFO:teuthology.orchestra.run.smithi101.stdout:(116/120): libselinux-devel-2.9-5.el8.x86_64.rp 11 MB/s | 200 kB 00:00 2023-06-22T17:17:09.909 INFO:teuthology.orchestra.run.smithi101.stdout:(117/120): libuuid-devel-2.32.1-27.el8.x86_64.r 7.0 MB/s | 97 kB 00:00 2023-06-22T17:17:09.910 INFO:teuthology.orchestra.run.smithi039.stdout:(28/120): ceph-mgr-cephadm-17.2.6-719.g42ddc46e 2.6 MB/s | 121 kB 00:00 2023-06-22T17:17:10.056 INFO:teuthology.orchestra.run.smithi039.stdout:(29/120): ceph-osd-17.2.6-719.g42ddc46e.el8.x86 2.7 MB/s | 17 MB 00:06 2023-06-22T17:17:10.310 INFO:teuthology.orchestra.run.smithi039.stdout:(30/120): ceph-mgr-dashboard-17.2.6-719.g42ddc4 4.2 MB/s | 1.7 MB 00:00 2023-06-22T17:17:10.357 INFO:teuthology.orchestra.run.smithi133.stdout:(119/120): librados2-17.2.6-719.g42ddc46e.el8.x 2.2 MB/s | 3.5 MB 00:01 2023-06-22T17:17:10.373 INFO:teuthology.orchestra.run.smithi039.stdout:(31/120): ceph-mgr-modules-core-17.2.6-719.g42d 3.7 MB/s | 238 kB 00:00 2023-06-22T17:17:10.431 INFO:teuthology.orchestra.run.smithi039.stdout:(32/120): ceph-mgr-rook-17.2.6-719.g42ddc46e.el 809 kB/s | 47 kB 00:00 2023-06-22T17:17:10.486 INFO:teuthology.orchestra.run.smithi039.stdout:(33/120): ceph-prometheus-alerts-17.2.6-719.g42 264 kB/s | 14 kB 00:00 2023-06-22T17:17:10.550 INFO:teuthology.orchestra.run.smithi039.stdout:(34/120): ceph-volume-17.2.6-719.g42ddc46e.el8. 4.0 MB/s | 264 kB 00:00 2023-06-22T17:17:10.615 INFO:teuthology.orchestra.run.smithi039.stdout:(35/120): cephadm-17.2.6-719.g42ddc46e.el8.noar 1.2 MB/s | 80 kB 00:00 2023-06-22T17:17:10.671 INFO:teuthology.orchestra.run.smithi039.stdout:(36/120): cephfs-top-17.2.6-719.g42ddc46e.el8.n 316 kB/s | 18 kB 00:00 2023-06-22T17:17:10.856 INFO:teuthology.orchestra.run.smithi101.stdout:(118/120): librbd1-17.2.6-719.g42ddc46e.el8.x86 3.8 MB/s | 3.6 MB 00:00 2023-06-22T17:17:11.203 INFO:teuthology.orchestra.run.smithi101.stdout:(119/120): librados2-17.2.6-719.g42ddc46e.el8.x 2.7 MB/s | 3.5 MB 00:01 2023-06-22T17:17:11.485 INFO:teuthology.orchestra.run.smithi039.stdout:(37/120): gperftools-libs-2.7-9.el8.x86_64.rpm 376 kB/s | 306 kB 00:00 2023-06-22T17:17:11.575 INFO:teuthology.orchestra.run.smithi039.stdout:(38/120): liboath-2.6.2-3.el8.x86_64.rpm 655 kB/s | 59 kB 00:00 2023-06-22T17:17:11.665 INFO:teuthology.orchestra.run.smithi039.stdout:(39/120): libunwind-1.3.1-3.el8.x86_64.rpm 833 kB/s | 75 kB 00:00 2023-06-22T17:17:12.033 INFO:teuthology.orchestra.run.smithi101.stdout:(120/120): ceph-test-17.2.6-719.g42ddc46e.el8.x 5.2 MB/s | 52 MB 00:09 2023-06-22T17:17:12.038 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-06-22T17:17:12.038 INFO:teuthology.orchestra.run.smithi101.stdout:Total 14 MB/s | 201 MB 00:13 2023-06-22T17:17:12.091 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-06-22T17:17:12.200 INFO:teuthology.orchestra.run.smithi039.stdout:(40/120): ceph-mgr-diskprediction-local-17.2.6- 3.4 MB/s | 7.4 MB 00:02 2023-06-22T17:17:12.208 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-06-22T17:17:12.208 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-06-22T17:17:12.412 INFO:teuthology.orchestra.run.smithi039.stdout:(41/120): python3-asyncssh-2.7.0-2.el8.noarch.r 638 kB/s | 476 kB 00:00 2023-06-22T17:17:12.681 INFO:teuthology.orchestra.run.smithi039.stdout:(42/120): python3-beautifulsoup4-4.6.3-2.el8.1. 688 kB/s | 185 kB 00:00 2023-06-22T17:17:12.757 INFO:teuthology.orchestra.run.smithi039.stdout:(43/120): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 78 kB/s | 44 kB 00:00 2023-06-22T17:17:12.775 INFO:teuthology.orchestra.run.smithi039.stdout:(44/120): python3-cachetools-3.1.1-4.el8.noarch 350 kB/s | 33 kB 00:00 2023-06-22T17:17:12.849 INFO:teuthology.orchestra.run.smithi039.stdout:(45/120): python3-certifi-2018.10.15-7.el8.noar 176 kB/s | 16 kB 00:00 2023-06-22T17:17:13.051 INFO:teuthology.orchestra.run.smithi039.stdout:(46/120): python3-cheroot-8.5.2-1.el8.noarch.rp 625 kB/s | 173 kB 00:00 2023-06-22T17:17:13.230 INFO:teuthology.orchestra.run.smithi039.stdout:(47/120): python3-google-auth-1.1.1-10.el8.noar 456 kB/s | 81 kB 00:00 2023-06-22T17:17:13.319 INFO:teuthology.orchestra.run.smithi039.stdout:(48/120): python3-jaraco-6.2-6.el8.noarch.rpm 124 kB/s | 11 kB 00:00 2023-06-22T17:17:13.392 INFO:teuthology.orchestra.run.smithi039.stdout:(49/120): python3-cherrypy-18.4.0-1.el8.noarch. 709 kB/s | 384 kB 00:00 2023-06-22T17:17:13.408 INFO:teuthology.orchestra.run.smithi039.stdout:(50/120): python3-jaraco-functools-2.0-4.el8.no 197 kB/s | 18 kB 00:00 2023-06-22T17:17:13.501 INFO:teuthology.orchestra.run.smithi039.stdout:(51/120): python3-logutils-0.3.5-11.el8.noarch. 530 kB/s | 49 kB 00:00 2023-06-22T17:17:13.590 INFO:teuthology.orchestra.run.smithi039.stdout:(52/120): python3-more-itertools-7.2.0-3.el8.no 655 kB/s | 59 kB 00:00 2023-06-22T17:17:13.645 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-06-22T17:17:13.649 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-06-22T17:17:13.681 INFO:teuthology.orchestra.run.smithi039.stdout:(53/120): python3-natsort-7.1.1-2.el8.noarch.rp 668 kB/s | 60 kB 00:00 2023-06-22T17:17:13.702 INFO:teuthology.orchestra.run.smithi133.stdout:(120/120): ceph-test-17.2.6-719.g42ddc46e.el8.x 3.6 MB/s | 52 MB 00:14 2023-06-22T17:17:13.707 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-06-22T17:17:13.707 INFO:teuthology.orchestra.run.smithi133.stdout:Total 12 MB/s | 201 MB 00:17 2023-06-22T17:17:13.760 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-06-22T17:17:13.877 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-06-22T17:17:13.877 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-06-22T17:17:14.127 INFO:teuthology.orchestra.run.smithi039.stdout:(54/120): python3-pecan-1.3.2-9.el8.noarch.rpm 634 kB/s | 283 kB 00:00 2023-06-22T17:17:14.217 INFO:teuthology.orchestra.run.smithi039.stdout:(55/120): python3-portend-2.6-1.el8.noarch.rpm 174 kB/s | 16 kB 00:00 2023-06-22T17:17:14.307 INFO:teuthology.orchestra.run.smithi039.stdout:(56/120): python3-repoze-lru-0.7-6.el8.noarch.r 375 kB/s | 34 kB 00:00 2023-06-22T17:17:14.959 INFO:teuthology.orchestra.run.smithi039.stdout:(57/120): ceph-test-17.2.6-719.g42ddc46e.el8.x8 6.8 MB/s | 52 MB 00:07 2023-06-22T17:17:14.961 INFO:teuthology.orchestra.run.smithi039.stdout:(58/120): python3-routes-2.4.1-12.el8.noarch.rp 300 kB/s | 196 kB 00:00 2023-06-22T17:17:15.008 INFO:teuthology.orchestra.run.smithi039.stdout:(59/120): python3-kubernetes-11.0.0-6.el8.noarc 739 kB/s | 1.2 MB 00:01 2023-06-22T17:17:15.055 INFO:teuthology.orchestra.run.smithi039.stdout:(60/120): python3-simplegeneric-0.8.1-17.el8.no 200 kB/s | 19 kB 00:00 2023-06-22T17:17:15.098 INFO:teuthology.orchestra.run.smithi039.stdout:(61/120): python3-singledispatch-3.4.0.3-18.el8 266 kB/s | 24 kB 00:00 2023-06-22T17:17:15.145 INFO:teuthology.orchestra.run.smithi039.stdout:(62/120): python3-tempora-1.14.1-5.el8.noarch.r 326 kB/s | 29 kB 00:00 2023-06-22T17:17:15.188 INFO:teuthology.orchestra.run.smithi039.stdout:(63/120): python3-trustme-0.6.0-4.el8.noarch.rp 307 kB/s | 27 kB 00:00 2023-06-22T17:17:15.307 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-06-22T17:17:15.312 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-06-22T17:17:15.504 INFO:teuthology.orchestra.run.smithi039.stdout:(64/120): python3-waitress-1.4.3-1.el8.noarch.r 712 kB/s | 255 kB 00:00 2023-06-22T17:17:15.507 INFO:teuthology.orchestra.run.smithi039.stdout:(65/120): python3-rsa-4.8-1.el8.noarch.rpm 112 kB/s | 61 kB 00:00 2023-06-22T17:17:15.553 INFO:teuthology.orchestra.run.smithi039.stdout:(66/120): python3-webob-1.8.5-1.el8.1.noarch.rp 689 kB/s | 251 kB 00:00 2023-06-22T17:17:15.595 INFO:teuthology.orchestra.run.smithi039.stdout:(67/120): python3-websocket-client-0.56.0-5.el8 678 kB/s | 61 kB 00:00 2023-06-22T17:17:15.649 INFO:teuthology.orchestra.run.smithi039.stdout:(68/120): python3-zc-lockfile-2.0-2.el8.noarch. 238 kB/s | 23 kB 00:00 2023-06-22T17:17:15.653 INFO:teuthology.orchestra.run.smithi039.stdout:(69/120): xmlstarlet-1.6.1-20.el8.x86_64.rpm 16 MB/s | 70 kB 00:00 2023-06-22T17:17:15.704 INFO:teuthology.orchestra.run.smithi039.stdout:(70/120): python3-webtest-2.0.33-1.el8.noarch.r 435 kB/s | 85 kB 00:00 2023-06-22T17:17:15.734 INFO:teuthology.orchestra.run.smithi039.stdout:(71/120): python3-prettytable-0.7.2-14.el8.noar 1.4 MB/s | 44 kB 00:00 2023-06-22T17:17:15.747 INFO:teuthology.orchestra.run.smithi039.stdout:(72/120): python3-jsonpatch-1.21-2.el8.noarch.r 2.1 MB/s | 27 kB 00:00 2023-06-22T17:17:15.763 INFO:teuthology.orchestra.run.smithi039.stdout:(73/120): jq-1.5-12.el8.x86_64.rpm 10 MB/s | 161 kB 00:00 2023-06-22T17:17:15.781 INFO:teuthology.orchestra.run.smithi039.stdout:(74/120): flex-2.6.1-9.el8.x86_64.rpm 18 MB/s | 320 kB 00:00 2023-06-22T17:17:15.801 INFO:teuthology.orchestra.run.smithi039.stdout:(75/120): python3-werkzeug-0.12.2-4.el8.noarch. 23 MB/s | 457 kB 00:00 2023-06-22T17:17:15.894 INFO:teuthology.orchestra.run.smithi039.stdout:(76/120): python3-babel-2.5.1-5.el8.noarch.rpm 51 MB/s | 4.8 MB 00:00 2023-06-22T17:17:16.012 INFO:teuthology.orchestra.run.smithi039.stdout:(77/120): python3-scipy-1.0.0-20.module+el8.1.0 41 MB/s | 15 MB 00:00 2023-06-22T17:17:16.013 INFO:teuthology.orchestra.run.smithi039.stdout:(78/120): python3-markupsafe-0.23-19.el8.x86_64 327 kB/s | 39 kB 00:00 2023-06-22T17:17:16.039 INFO:teuthology.orchestra.run.smithi039.stdout:(79/120): lttng-ust-2.8.1-11.el8.x86_64.rpm 10 MB/s | 259 kB 00:00 2023-06-22T17:17:16.042 INFO:teuthology.orchestra.run.smithi039.stdout:(80/120): bison-3.0.4-10.el8.x86_64.rpm 22 MB/s | 688 kB 00:00 2023-06-22T17:17:16.056 INFO:teuthology.orchestra.run.smithi039.stdout:(81/120): python3-numpy-f2py-1.14.3-9.el8.x86_6 13 MB/s | 225 kB 00:00 2023-06-22T17:17:16.065 INFO:teuthology.orchestra.run.smithi039.stdout:(82/120): python3-pyasn1-0.3.7-6.el8.noarch.rpm 5.5 MB/s | 126 kB 00:00 2023-06-22T17:17:16.081 INFO:teuthology.orchestra.run.smithi039.stdout:(83/120): oniguruma-6.8.2-2.el8.x86_64.rpm 12 MB/s | 187 kB 00:00 2023-06-22T17:17:16.102 INFO:teuthology.orchestra.run.smithi039.stdout:(84/120): python3-jinja2-2.10.1-2.el8_0.noarch. 25 MB/s | 538 kB 00:00 2023-06-22T17:17:16.126 INFO:teuthology.orchestra.run.smithi039.stdout:(85/120): python3-numpy-1.14.3-9.el8.x86_64.rpm 54 MB/s | 3.7 MB 00:00 2023-06-22T17:17:16.129 INFO:teuthology.orchestra.run.smithi039.stdout:(86/120): socat-1.7.3.3-2.el8.x86_64.rpm 11 MB/s | 302 kB 00:00 2023-06-22T17:17:16.142 INFO:teuthology.orchestra.run.smithi039.stdout:(87/120): python3-mako-1.0.6-13.el8.noarch.rpm 9.5 MB/s | 157 kB 00:00 2023-06-22T17:17:16.153 INFO:teuthology.orchestra.run.smithi039.stdout:(88/120): python3-pyasn1-modules-0.3.7-6.el8.no 4.6 MB/s | 110 kB 00:00 2023-06-22T17:17:16.162 INFO:teuthology.orchestra.run.smithi039.stdout:(89/120): librdkafka-0.11.4-1.el8.x86_64.rpm 18 MB/s | 353 kB 00:00 2023-06-22T17:17:16.172 INFO:teuthology.orchestra.run.smithi039.stdout:(90/120): python3-jsonpointer-1.10-11.el8.noarc 1.0 MB/s | 20 kB 00:00 2023-06-22T17:17:16.189 INFO:teuthology.orchestra.run.smithi039.stdout:(91/120): python3-pytz-2017.2-9.el8.noarch.rpm 3.2 MB/s | 54 kB 00:00 2023-06-22T17:17:16.193 INFO:teuthology.orchestra.run.smithi039.stdout:(92/120): libtool-2.4.6-25.el8.x86_64.rpm 23 MB/s | 709 kB 00:00 2023-06-22T17:17:16.284 INFO:teuthology.orchestra.run.smithi039.stdout:(93/120): openblas-0.3.12-1.el8.x86_64.rpm 51 MB/s | 4.6 MB 00:00 2023-06-22T17:17:16.299 INFO:teuthology.orchestra.run.smithi039.stdout:(94/120): openblas-threads-0.3.12-1.el8.x86_64. 43 MB/s | 4.7 MB 00:00 2023-06-22T17:17:16.309 INFO:teuthology.orchestra.run.smithi039.stdout:(95/120): automake-1.16.1-7.el8.noarch.rpm 28 MB/s | 713 kB 00:00 2023-06-22T17:17:16.316 INFO:teuthology.orchestra.run.smithi039.stdout:(96/120): libaio-devel-0.3.112-1.el8.x86_64.rpm 1.1 MB/s | 19 kB 00:00 2023-06-22T17:17:16.325 INFO:teuthology.orchestra.run.smithi039.stdout:(97/120): python3-jwt-1.6.1-2.el8.noarch.rpm 2.6 MB/s | 43 kB 00:00 2023-06-22T17:17:16.334 INFO:teuthology.orchestra.run.smithi039.stdout:(98/120): python3-oauthlib-2.1.0-1.el8.noarch.r 8.5 MB/s | 155 kB 00:00 2023-06-22T17:17:16.344 INFO:teuthology.orchestra.run.smithi039.stdout:(99/120): pcre2-utf32-10.32-2.el8.x86_64.rpm 12 MB/s | 220 kB 00:00 2023-06-22T17:17:16.347 INFO:teuthology.orchestra.run.smithi039.stdout:(100/120): keyutils-libs-devel-1.5.10-6.el8.x86 3.7 MB/s | 48 kB 00:00 2023-06-22T17:17:16.361 INFO:teuthology.orchestra.run.smithi039.stdout:(101/120): pcre2-utf16-10.32-2.el8.x86_64.rpm 13 MB/s | 229 kB 00:00 2023-06-22T17:17:16.364 INFO:teuthology.orchestra.run.smithi039.stdout:(102/120): python3-requests-oauthlib-1.0.0-1.el 2.4 MB/s | 43 kB 00:00 2023-06-22T17:17:16.381 INFO:teuthology.orchestra.run.smithi039.stdout:(103/120): libverto-devel-0.3.0-5.el8.x86_64.rp 1.0 MB/s | 18 kB 00:00 2023-06-22T17:17:16.408 INFO:teuthology.orchestra.run.smithi039.stdout:(104/120): openssl-devel-1.1.1g-15.el8_3.x86_64 49 MB/s | 2.3 MB 00:00 2023-06-22T17:17:16.412 INFO:teuthology.orchestra.run.smithi039.stdout:(105/120): pcre2-devel-10.32-2.el8.x86_64.rpm 20 MB/s | 605 kB 00:00 2023-06-22T17:17:16.426 INFO:teuthology.orchestra.run.smithi039.stdout:(106/120): fuse-2.9.7-12.el8.x86_64.rpm 4.4 MB/s | 83 kB 00:00 2023-06-22T17:17:16.428 INFO:teuthology.orchestra.run.smithi039.stdout:(107/120): elfutils-libelf-devel-0.182-3.el8.x8 3.6 MB/s | 59 kB 00:00 2023-06-22T17:17:16.443 INFO:teuthology.orchestra.run.smithi039.stdout:(108/120): libcom_err-devel-1.45.6-1.el8.x86_64 2.4 MB/s | 38 kB 00:00 2023-06-22T17:17:16.447 INFO:teuthology.orchestra.run.smithi039.stdout:(109/120): libquadmath-8.4.1-1.el8.x86_64.rpm 8.2 MB/s | 169 kB 00:00 2023-06-22T17:17:16.464 INFO:teuthology.orchestra.run.smithi039.stdout:(110/120): libsepol-devel-2.9-2.el8.x86_64.rpm 4.1 MB/s | 86 kB 00:00 2023-06-22T17:17:16.472 INFO:teuthology.orchestra.run.smithi039.stdout:(111/120): zlib-devel-1.2.11-17.el8.x86_64.rpm 2.2 MB/s | 58 kB 00:00 2023-06-22T17:17:16.481 INFO:teuthology.orchestra.run.smithi039.stdout:(112/120): librabbitmq-0.9.0-3.el8.x86_64.rpm 2.7 MB/s | 47 kB 00:00 2023-06-22T17:17:16.495 INFO:teuthology.orchestra.run.smithi039.stdout:(113/120): krb5-devel-1.18.2-8.3.el8_4.x86_64.r 24 MB/s | 559 kB 00:00 2023-06-22T17:17:16.506 INFO:teuthology.orchestra.run.smithi039.stdout:(114/120): libgfortran-8.4.1-1.el8.x86_64.rpm 26 MB/s | 639 kB 00:00 2023-06-22T17:17:16.513 INFO:teuthology.orchestra.run.smithi039.stdout:(115/120): xfsprogs-devel-5.0.0-8.el8.x86_64.rp 4.0 MB/s | 70 kB 00:00 2023-06-22T17:17:16.527 INFO:teuthology.orchestra.run.smithi039.stdout:(116/120): libselinux-devel-2.9-5.el8.x86_64.rp 9.1 MB/s | 200 kB 00:00 2023-06-22T17:17:16.537 INFO:teuthology.orchestra.run.smithi039.stdout:(117/120): libuuid-devel-2.32.1-27.el8.x86_64.r 3.9 MB/s | 97 kB 00:00 2023-06-22T17:17:17.727 INFO:teuthology.orchestra.run.smithi039.stdout:(118/120): librados2-17.2.6-719.g42ddc46e.el8.x 2.9 MB/s | 3.5 MB 00:01 2023-06-22T17:17:18.030 INFO:teuthology.orchestra.run.smithi039.stdout:(119/120): librbd1-17.2.6-719.g42ddc46e.el8.x86 2.4 MB/s | 3.6 MB 00:01 2023-06-22T17:17:18.134 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-06-22T17:17:18.148 INFO:teuthology.orchestra.run.smithi039.stdout:(120/120): thrift-0.13.0-2.el8.x86_64.rpm 695 kB/s | 1.7 MB 00:02 2023-06-22T17:17:18.154 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-06-22T17:17:18.155 INFO:teuthology.orchestra.run.smithi039.stdout:Total 12 MB/s | 201 MB 00:17 2023-06-22T17:17:18.213 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-06-22T17:17:18.312 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/122 2023-06-22T17:17:18.342 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-06-22T17:17:18.342 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-06-22T17:17:18.529 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 2/122 2023-06-22T17:17:18.734 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 3/122 2023-06-22T17:17:18.866 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 3/122 2023-06-22T17:17:19.084 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/122 2023-06-22T17:17:19.490 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/122 2023-06-22T17:17:19.684 INFO:teuthology.orchestra.run.smithi101.stdout: Upgrading : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 5/122 2023-06-22T17:17:19.856 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-06-22T17:17:19.861 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-06-22T17:17:20.101 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 5/122 2023-06-22T17:17:20.266 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-06-22T17:17:20.334 INFO:teuthology.orchestra.run.smithi101.stdout: Upgrading : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-22T17:17:20.453 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-22T17:17:20.476 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/122 2023-06-22T17:17:20.642 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-22T17:17:20.678 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 2/122 2023-06-22T17:17:20.728 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-22T17:17:20.776 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 2/122 2023-06-22T17:17:21.011 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/122 2023-06-22T17:17:21.099 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 8/122 2023-06-22T17:17:21.381 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/122 2023-06-22T17:17:21.800 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/122 2023-06-22T17:17:21.976 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-22T17:17:22.011 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 4/122 2023-06-22T17:17:22.119 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-22T17:17:22.354 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/122 2023-06-22T17:17:22.414 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 4/122 2023-06-22T17:17:22.529 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/122 2023-06-22T17:17:22.602 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 5/122 2023-06-22T17:17:22.731 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 5/122 2023-06-22T17:17:22.759 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-22T17:17:22.855 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-22T17:17:22.952 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-22T17:17:23.017 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 14/122 2023-06-22T17:17:23.051 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-22T17:17:23.081 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 14/122 2023-06-22T17:17:23.249 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-22T17:17:23.263 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 15/122 2023-06-22T17:17:23.401 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 16/122 2023-06-22T17:17:23.535 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 16/122 2023-06-22T17:17:23.557 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/122 2023-06-22T17:17:23.700 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 17/122 2023-06-22T17:17:24.227 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/122 2023-06-22T17:17:24.266 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 17/122 2023-06-22T17:17:24.426 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-22T17:17:24.483 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 18/122 2023-06-22T17:17:24.524 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-22T17:17:24.731 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/122 2023-06-22T17:17:24.908 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/122 2023-06-22T17:17:25.016 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 18/122 2023-06-22T17:17:25.118 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-22T17:17:25.217 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : openblas-0.3.12-1.el8.x86_64 19/122 2023-06-22T17:17:25.225 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-22T17:17:25.393 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 14/122 2023-06-22T17:17:25.411 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-06-22T17:17:25.470 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 14/122 2023-06-22T17:17:25.608 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/122 2023-06-22T17:17:25.630 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 15/122 2023-06-22T17:17:25.805 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 2/122 2023-06-22T17:17:25.906 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 16/122 2023-06-22T17:17:25.931 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 19/122 2023-06-22T17:17:25.977 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 3/122 2023-06-22T17:17:26.083 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 3/122 2023-06-22T17:17:26.116 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 17/122 2023-06-22T17:17:26.202 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 20/122 2023-06-22T17:17:26.267 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 18/122 2023-06-22T17:17:26.343 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/122 2023-06-22T17:17:26.373 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 21/122 2023-06-22T17:17:26.435 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 19/122 2023-06-22T17:17:26.545 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 22/122 2023-06-22T17:17:26.565 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 19/122 2023-06-22T17:17:26.718 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 20/122 2023-06-22T17:17:26.751 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/122 2023-06-22T17:17:26.803 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/122 2023-06-22T17:17:26.952 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 5/122 2023-06-22T17:17:27.101 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 24/122 2023-06-22T17:17:27.258 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 20/122 2023-06-22T17:17:27.361 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/122 2023-06-22T17:17:27.378 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 5/122 2023-06-22T17:17:27.460 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 21/122 2023-06-22T17:17:27.531 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 26/122 2023-06-22T17:17:27.577 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-22T17:17:27.670 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 27/122 2023-06-22T17:17:27.711 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-22T17:17:27.897 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 28/122 2023-06-22T17:17:27.952 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-22T17:17:27.983 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 21/122 2023-06-22T17:17:28.054 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-22T17:17:28.176 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : openblas-0.3.12-1.el8.x86_64 22/122 2023-06-22T17:17:28.408 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 8/122 2023-06-22T17:17:28.878 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 29/122 2023-06-22T17:17:28.897 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 22/122 2023-06-22T17:17:29.125 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/122 2023-06-22T17:17:29.179 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 23/122 2023-06-22T17:17:29.202 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 30/122 2023-06-22T17:17:29.276 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-22T17:17:29.409 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-22T17:17:29.460 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 24/122 2023-06-22T17:17:29.498 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 31/122 2023-06-22T17:17:29.606 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/122 2023-06-22T17:17:29.695 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/122 2023-06-22T17:17:29.739 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 32/122 2023-06-22T17:17:29.786 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/122 2023-06-22T17:17:29.874 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 26/122 2023-06-22T17:17:29.968 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-22T17:17:29.982 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 33/122 2023-06-22T17:17:30.029 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 27/122 2023-06-22T17:17:30.082 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-22T17:17:30.225 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 34/122 2023-06-22T17:17:30.261 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 28/122 2023-06-22T17:17:30.276 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 14/122 2023-06-22T17:17:30.347 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 14/122 2023-06-22T17:17:30.495 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 15/122 2023-06-22T17:17:30.498 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 29/122 2023-06-22T17:17:30.627 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 16/122 2023-06-22T17:17:30.767 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 16/122 2023-06-22T17:17:30.800 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 30/122 2023-06-22T17:17:30.951 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 17/122 2023-06-22T17:17:31.026 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 31/122 2023-06-22T17:17:31.542 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 17/122 2023-06-22T17:17:31.743 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 18/122 2023-06-22T17:17:32.278 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 34/122 2023-06-22T17:17:32.309 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 18/122 2023-06-22T17:17:32.501 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openblas-0.3.12-1.el8.x86_64 19/122 2023-06-22T17:17:32.570 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 35/122 2023-06-22T17:17:32.804 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-rsa-4.8-1.el8.noarch 36/122 2023-06-22T17:17:33.002 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 37/122 2023-06-22T17:17:33.073 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 31/122 2023-06-22T17:17:33.225 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 38/122 2023-06-22T17:17:33.246 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 19/122 2023-06-22T17:17:33.423 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 32/122 2023-06-22T17:17:33.428 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 39/122 2023-06-22T17:17:33.497 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 20/122 2023-06-22T17:17:33.615 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 40/122 2023-06-22T17:17:33.684 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 21/122 2023-06-22T17:17:33.797 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libuuid-devel-2.32.1-27.el8.x86_64 41/122 2023-06-22T17:17:33.881 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 22/122 2023-06-22T17:17:34.003 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libsepol-devel-2.9-2.el8.x86_64 42/122 2023-06-22T17:17:34.162 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/122 2023-06-22T17:17:34.176 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 43/122 2023-06-22T17:17:34.335 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : fuse-2.9.7-12.el8.x86_64 44/122 2023-06-22T17:17:34.417 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/122 2023-06-22T17:17:34.448 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 24/122 2023-06-22T17:17:34.503 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 45/122 2023-06-22T17:17:34.673 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/122 2023-06-22T17:17:34.688 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 46/122 2023-06-22T17:17:34.745 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 34/122 2023-06-22T17:17:34.827 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 47/122 2023-06-22T17:17:34.883 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 26/122 2023-06-22T17:17:34.995 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 35/122 2023-06-22T17:17:35.021 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 27/122 2023-06-22T17:17:35.179 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 48/122 2023-06-22T17:17:35.205 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rsa-4.8-1.el8.noarch 36/122 2023-06-22T17:17:35.253 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 28/122 2023-06-22T17:17:35.395 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 37/122 2023-06-22T17:17:35.445 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 49/122 2023-06-22T17:17:35.626 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 38/122 2023-06-22T17:17:35.741 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 50/122 2023-06-22T17:17:35.813 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 39/122 2023-06-22T17:17:36.080 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : krb5-devel-1.18.2-8.3.el8_4.x86_64 51/122 2023-06-22T17:17:36.090 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 40/122 2023-06-22T17:17:36.306 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 29/122 2023-06-22T17:17:36.322 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : automake-1.16.1-7.el8.noarch 52/122 2023-06-22T17:17:36.332 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : automake-1.16.1-7.el8.noarch 41/122 2023-06-22T17:17:36.489 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 53/122 2023-06-22T17:17:36.515 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 42/122 2023-06-22T17:17:36.655 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 30/122 2023-06-22T17:17:36.689 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 54/122 2023-06-22T17:17:36.728 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 43/122 2023-06-22T17:17:36.905 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 55/122 2023-06-22T17:17:36.911 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 44/122 2023-06-22T17:17:36.938 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 31/122 2023-06-22T17:17:37.074 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 56/122 2023-06-22T17:17:37.108 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 45/122 2023-06-22T17:17:37.156 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 56/122 2023-06-22T17:17:37.183 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 45/122 2023-06-22T17:17:37.191 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 32/122 2023-06-22T17:17:37.315 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : jq-1.5-12.el8.x86_64 57/122 2023-06-22T17:17:37.349 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : jq-1.5-12.el8.x86_64 46/122 2023-06-22T17:17:37.471 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 33/122 2023-06-22T17:17:37.513 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 58/122 2023-06-22T17:17:37.515 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 47/122 2023-06-22T17:17:37.666 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 59/122 2023-06-22T17:17:37.684 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 34/122 2023-06-22T17:17:37.723 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libuuid-devel-2.32.1-27.el8.x86_64 48/122 2023-06-22T17:17:37.847 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 60/122 2023-06-22T17:17:37.895 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libsepol-devel-2.9-2.el8.x86_64 49/122 2023-06-22T17:17:38.059 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 50/122 2023-06-22T17:17:38.131 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 61/122 2023-06-22T17:17:38.235 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : fuse-2.9.7-12.el8.x86_64 51/122 2023-06-22T17:17:38.365 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 62/122 2023-06-22T17:17:38.418 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 52/122 2023-06-22T17:17:38.556 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 63/122 2023-06-22T17:17:38.630 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 53/122 2023-06-22T17:17:38.711 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 64/122 2023-06-22T17:17:38.786 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 54/122 2023-06-22T17:17:38.843 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 65/122 2023-06-22T17:17:39.016 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 66/122 2023-06-22T17:17:39.090 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/122 2023-06-22T17:17:39.218 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 67/122 2023-06-22T17:17:39.327 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/122 2023-06-22T17:17:39.425 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 68/122 2023-06-22T17:17:39.593 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/122 2023-06-22T17:17:39.602 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 69/122 2023-06-22T17:17:39.779 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 70/122 2023-06-22T17:17:39.865 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 34/122 2023-06-22T17:17:39.888 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : krb5-devel-1.18.2-8.3.el8_4.x86_64 58/122 2023-06-22T17:17:39.992 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 71/122 2023-06-22T17:17:40.041 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 59/122 2023-06-22T17:17:40.218 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 72/122 2023-06-22T17:17:40.220 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 60/122 2023-06-22T17:17:40.306 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 35/122 2023-06-22T17:17:40.412 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 73/122 2023-06-22T17:17:40.453 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 61/122 2023-06-22T17:17:40.516 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rsa-4.8-1.el8.noarch 36/122 2023-06-22T17:17:40.674 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 62/122 2023-06-22T17:17:40.684 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-portend-2.6-1.el8.noarch 74/122 2023-06-22T17:17:40.704 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 37/122 2023-06-22T17:17:40.825 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 63/122 2023-06-22T17:17:40.938 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 38/122 2023-06-22T17:17:40.947 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 75/122 2023-06-22T17:17:40.987 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 64/122 2023-06-22T17:17:41.114 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 39/122 2023-06-22T17:17:41.115 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 76/122 2023-06-22T17:17:41.126 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 65/122 2023-06-22T17:17:41.299 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 40/122 2023-06-22T17:17:41.354 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 77/122 2023-06-22T17:17:41.358 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 66/122 2023-06-22T17:17:41.466 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libuuid-devel-2.32.1-27.el8.x86_64 41/122 2023-06-22T17:17:41.552 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 67/122 2023-06-22T17:17:41.654 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libsepol-devel-2.9-2.el8.x86_64 42/122 2023-06-22T17:17:41.759 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 68/122 2023-06-22T17:17:41.801 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 43/122 2023-06-22T17:17:41.944 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : fuse-2.9.7-12.el8.x86_64 44/122 2023-06-22T17:17:41.969 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 69/122 2023-06-22T17:17:42.080 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 45/122 2023-06-22T17:17:42.121 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 70/122 2023-06-22T17:17:42.264 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 46/122 2023-06-22T17:17:42.344 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 71/122 2023-06-22T17:17:42.454 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 47/122 2023-06-22T17:17:42.475 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 78/122 2023-06-22T17:17:42.560 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 72/122 2023-06-22T17:17:42.768 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 79/122 2023-06-22T17:17:42.769 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 48/122 2023-06-22T17:17:42.771 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 73/122 2023-06-22T17:17:43.008 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 80/122 2023-06-22T17:17:43.026 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-portend-2.6-1.el8.noarch 74/122 2023-06-22T17:17:43.028 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 49/122 2023-06-22T17:17:43.264 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 75/122 2023-06-22T17:17:43.298 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 81/122 2023-06-22T17:17:43.351 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 50/122 2023-06-22T17:17:43.472 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 76/122 2023-06-22T17:17:43.579 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 82/122 2023-06-22T17:17:43.680 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : krb5-devel-1.18.2-8.3.el8_4.x86_64 51/122 2023-06-22T17:17:43.698 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 77/122 2023-06-22T17:17:43.861 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : automake-1.16.1-7.el8.noarch 52/122 2023-06-22T17:17:43.880 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 83/122 2023-06-22T17:17:44.049 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 53/122 2023-06-22T17:17:44.160 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 84/122 2023-06-22T17:17:44.255 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 54/122 2023-06-22T17:17:44.332 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 85/122 2023-06-22T17:17:44.446 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 55/122 2023-06-22T17:17:44.590 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 86/122 2023-06-22T17:17:44.608 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 56/122 2023-06-22T17:17:44.689 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 56/122 2023-06-22T17:17:44.821 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 78/122 2023-06-22T17:17:44.883 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : jq-1.5-12.el8.x86_64 57/122 2023-06-22T17:17:45.069 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 58/122 2023-06-22T17:17:45.129 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 79/122 2023-06-22T17:17:45.225 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 59/122 2023-06-22T17:17:45.357 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 80/122 2023-06-22T17:17:45.414 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 60/122 2023-06-22T17:17:45.674 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 81/122 2023-06-22T17:17:45.692 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 61/122 2023-06-22T17:17:45.933 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 82/122 2023-06-22T17:17:46.000 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 62/122 2023-06-22T17:17:46.154 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 63/122 2023-06-22T17:17:46.279 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 83/122 2023-06-22T17:17:46.346 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 64/122 2023-06-22T17:17:46.494 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 65/122 2023-06-22T17:17:46.526 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 84/122 2023-06-22T17:17:46.694 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 66/122 2023-06-22T17:17:46.708 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 85/122 2023-06-22T17:17:46.888 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 67/122 2023-06-22T17:17:46.932 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 86/122 2023-06-22T17:17:47.105 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 68/122 2023-06-22T17:17:47.295 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 69/122 2023-06-22T17:17:47.457 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 70/122 2023-06-22T17:17:47.691 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 71/122 2023-06-22T17:17:47.914 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 72/122 2023-06-22T17:17:48.123 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 73/122 2023-06-22T17:17:48.413 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-portend-2.6-1.el8.noarch 74/122 2023-06-22T17:17:48.666 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 75/122 2023-06-22T17:17:48.859 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 76/122 2023-06-22T17:17:49.089 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 77/122 2023-06-22T17:17:49.466 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-22T17:17:49.656 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-22T17:17:50.249 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-22T17:17:50.268 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 78/122 2023-06-22T17:17:50.547 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 88/122 2023-06-22T17:17:50.553 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 79/122 2023-06-22T17:17:50.654 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 88/122 2023-06-22T17:17:50.768 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 80/122 2023-06-22T17:17:50.797 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 89/122 2023-06-22T17:17:51.090 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 81/122 2023-06-22T17:17:51.333 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 82/122 2023-06-22T17:17:51.647 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 83/122 2023-06-22T17:17:51.791 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-22T17:17:51.884 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 84/122 2023-06-22T17:17:52.014 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-22T17:17:52.067 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 85/122 2023-06-22T17:17:52.249 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 86/122 2023-06-22T17:17:52.616 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-22T17:17:53.293 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 88/122 2023-06-22T17:17:53.401 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 88/122 2023-06-22T17:17:53.536 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 89/122 2023-06-22T17:17:57.803 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-22T17:17:58.031 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-22T17:17:58.648 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-22T17:17:59.553 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 88/122 2023-06-22T17:17:59.662 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 88/122 2023-06-22T17:17:59.801 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 89/122 2023-06-22T17:18:08.411 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 89/122 2023-06-22T17:18:08.412 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /sys 2023-06-22T17:18:08.412 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /proc 2023-06-22T17:18:08.412 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /dev 2023-06-22T17:18:08.412 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /run 2023-06-22T17:18:08.412 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /mnt 2023-06-22T17:18:08.412 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /var/tmp 2023-06-22T17:18:08.413 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /home 2023-06-22T17:18:08.413 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /tmp 2023-06-22T17:18:08.413 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /dev 2023-06-22T17:18:08.413 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-06-22T17:18:08.753 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 90/122 2023-06-22T17:18:10.235 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 90/122 2023-06-22T17:18:10.395 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 91/122 2023-06-22T17:18:10.617 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 91/122 2023-06-22T17:18:10.786 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 92/122 2023-06-22T17:18:11.177 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 92/122 2023-06-22T17:18:11.232 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 89/122 2023-06-22T17:18:11.233 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /sys 2023-06-22T17:18:11.233 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /proc 2023-06-22T17:18:11.233 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /dev 2023-06-22T17:18:11.233 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /run 2023-06-22T17:18:11.233 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /mnt 2023-06-22T17:18:11.233 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /var/tmp 2023-06-22T17:18:11.234 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /home 2023-06-22T17:18:11.234 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /tmp 2023-06-22T17:18:11.234 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /dev 2023-06-22T17:18:11.234 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-06-22T17:18:11.353 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 93/122 2023-06-22T17:18:11.369 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 93/122 2023-06-22T17:18:11.602 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 90/122 2023-06-22T17:18:13.085 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 90/122 2023-06-22T17:18:13.232 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 91/122 2023-06-22T17:18:13.453 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 91/122 2023-06-22T17:18:13.635 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 92/122 2023-06-22T17:18:13.991 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 94/122 2023-06-22T17:18:14.027 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 92/122 2023-06-22T17:18:14.185 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 93/122 2023-06-22T17:18:14.202 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 93/122 2023-06-22T17:18:14.227 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 94/122 2023-06-22T17:18:14.334 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 95/122 2023-06-22T17:18:14.515 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 96/122 2023-06-22T17:18:14.710 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 97/122 2023-06-22T17:18:14.924 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 97/122 2023-06-22T17:18:15.127 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 98/122 2023-06-22T17:18:16.095 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 98/122 2023-06-22T17:18:16.260 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 99/122 2023-06-22T17:18:16.358 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 99/122 2023-06-22T17:18:16.758 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 100/122 2023-06-22T17:18:16.893 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 94/122 2023-06-22T17:18:16.969 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 101/122 2023-06-22T17:18:17.006 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 101/122 2023-06-22T17:18:17.042 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 94/122 2023-06-22T17:18:17.168 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 102/122 2023-06-22T17:18:17.179 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 102/122 2023-06-22T17:18:17.266 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 95/122 2023-06-22T17:18:17.448 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 96/122 2023-06-22T17:18:17.592 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 97/122 2023-06-22T17:18:17.806 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 97/122 2023-06-22T17:18:17.980 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 98/122 2023-06-22T17:18:18.338 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 89/122 2023-06-22T17:18:18.338 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /sys 2023-06-22T17:18:18.338 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /proc 2023-06-22T17:18:18.339 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /dev 2023-06-22T17:18:18.339 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /run 2023-06-22T17:18:18.339 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /mnt 2023-06-22T17:18:18.339 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /var/tmp 2023-06-22T17:18:18.339 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /home 2023-06-22T17:18:18.339 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /tmp 2023-06-22T17:18:18.339 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /dev 2023-06-22T17:18:18.339 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:18:18.728 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 90/122 2023-06-22T17:18:18.803 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 103/122 2023-06-22T17:18:18.949 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 98/122 2023-06-22T17:18:18.995 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 104/122 2023-06-22T17:18:19.084 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 104/122 2023-06-22T17:18:19.109 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 99/122 2023-06-22T17:18:19.207 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 99/122 2023-06-22T17:18:19.269 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 105/122 2023-06-22T17:18:19.311 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 105/122 2023-06-22T17:18:19.493 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 106/122 2023-06-22T17:18:19.560 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 100/122 2023-06-22T17:18:19.738 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 101/122 2023-06-22T17:18:19.775 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 101/122 2023-06-22T17:18:19.841 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 106/122 2023-06-22T17:18:19.958 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 102/122 2023-06-22T17:18:19.970 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 102/122 2023-06-22T17:18:19.993 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 107/122 2023-06-22T17:18:20.321 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 90/122 2023-06-22T17:18:20.457 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 91/122 2023-06-22T17:18:20.691 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 91/122 2023-06-22T17:18:20.836 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 92/122 2023-06-22T17:18:21.250 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 92/122 2023-06-22T17:18:21.394 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 93/122 2023-06-22T17:18:21.411 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 93/122 2023-06-22T17:18:21.597 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 103/122 2023-06-22T17:18:21.797 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 104/122 2023-06-22T17:18:21.893 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 104/122 2023-06-22T17:18:22.076 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 105/122 2023-06-22T17:18:22.118 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 105/122 2023-06-22T17:18:22.292 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 106/122 2023-06-22T17:18:22.641 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 106/122 2023-06-22T17:18:22.850 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 107/122 2023-06-22T17:18:24.344 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 94/122 2023-06-22T17:18:24.477 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 94/122 2023-06-22T17:18:24.675 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 95/122 2023-06-22T17:18:24.882 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 96/122 2023-06-22T17:18:24.934 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 107/122 2023-06-22T17:18:25.052 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 97/122 2023-06-22T17:18:25.245 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 108/122 2023-06-22T17:18:25.277 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 97/122 2023-06-22T17:18:25.465 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 98/122 2023-06-22T17:18:25.467 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libtool-2.4.6-25.el8.x86_64 109/122 2023-06-22T17:18:26.171 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 109/122 2023-06-22T17:18:26.500 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 98/122 2023-06-22T17:18:26.562 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : openssl-devel-1:1.1.1g-15.el8_3.x86_64 110/122 2023-06-22T17:18:26.660 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 99/122 2023-06-22T17:18:26.746 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 111/122 2023-06-22T17:18:26.761 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 99/122 2023-06-22T17:18:26.887 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : xfsprogs-devel-5.0.0-8.el8.x86_64 112/122 2023-06-22T17:18:27.030 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 113/122 2023-06-22T17:18:27.104 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 100/122 2023-06-22T17:18:27.197 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : elfutils-libelf-devel-0.182-3.el8.x86_64 114/122 2023-06-22T17:18:27.282 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 101/122 2023-06-22T17:18:27.319 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 101/122 2023-06-22T17:18:27.350 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 115/122 2023-06-22T17:18:27.527 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 116/122 2023-06-22T17:18:27.634 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 102/122 2023-06-22T17:18:27.646 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 102/122 2023-06-22T17:18:27.701 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 117/122 2023-06-22T17:18:27.774 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 107/122 2023-06-22T17:18:27.949 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 118/122 2023-06-22T17:18:28.125 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : bison-3.0.4-10.el8.x86_64 119/122 2023-06-22T17:18:28.178 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 119/122 2023-06-22T17:18:28.375 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : flex-2.6.1-9.el8.x86_64 120/122 2023-06-22T17:18:28.384 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 120/122 2023-06-22T17:18:28.384 INFO:teuthology.orchestra.run.smithi101.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-06-22T17:18:28.603 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-06-22T17:18:28.603 INFO:teuthology.orchestra.run.smithi101.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 122/122 2023-06-22T17:18:28.708 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 108/122 2023-06-22T17:18:29.056 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : openssl-devel-1:1.1.1g-15.el8_3.x86_64 109/122 2023-06-22T17:18:29.175 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 122/122 2023-06-22T17:18:29.175 INFO:teuthology.orchestra.run.smithi101.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-06-22T17:18:29.175 INFO:teuthology.orchestra.run.smithi101.stdout:All rules containing unresolvable specifiers will be skipped. 2023-06-22T17:18:29.176 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-06-22T17:18:29.244 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 110/122 2023-06-22T17:18:29.335 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 103/122 2023-06-22T17:18:29.445 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : xfsprogs-devel-5.0.0-8.el8.x86_64 111/122 2023-06-22T17:18:29.534 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 1/122 2023-06-22T17:18:29.535 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2/122 2023-06-22T17:18:29.535 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 3/122 2023-06-22T17:18:29.535 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 4/122 2023-06-22T17:18:29.535 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 5/122 2023-06-22T17:18:29.535 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-22T17:18:29.535 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-22T17:18:29.535 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 8/122 2023-06-22T17:18:29.536 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 9/122 2023-06-22T17:18:29.537 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-22T17:18:29.537 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 11/122 2023-06-22T17:18:29.537 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 12/122 2023-06-22T17:18:29.538 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-22T17:18:29.538 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 14/122 2023-06-22T17:18:29.538 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 15/122 2023-06-22T17:18:29.538 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 16/122 2023-06-22T17:18:29.538 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 17/122 2023-06-22T17:18:29.538 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 18/122 2023-06-22T17:18:29.538 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 19/122 2023-06-22T17:18:29.539 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 20/122 2023-06-22T17:18:29.539 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 21/122 2023-06-22T17:18:29.539 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 22/122 2023-06-22T17:18:29.539 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 23/122 2023-06-22T17:18:29.539 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 24/122 2023-06-22T17:18:29.539 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 25/122 2023-06-22T17:18:29.549 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 26/122 2023-06-22T17:18:29.549 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 27/122 2023-06-22T17:18:29.549 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 28/122 2023-06-22T17:18:29.550 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 29/122 2023-06-22T17:18:29.550 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 30/122 2023-06-22T17:18:29.550 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 31/122 2023-06-22T17:18:29.550 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 32/122 2023-06-22T17:18:29.550 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 33/122 2023-06-22T17:18:29.550 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 34/122 2023-06-22T17:18:29.551 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 35/122 2023-06-22T17:18:29.551 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 36/122 2023-06-22T17:18:29.551 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 37/122 2023-06-22T17:18:29.551 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 38/122 2023-06-22T17:18:29.551 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/122 2023-06-22T17:18:29.551 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/122 2023-06-22T17:18:29.551 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/122 2023-06-22T17:18:29.551 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/122 2023-06-22T17:18:29.552 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/122 2023-06-22T17:18:29.552 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/122 2023-06-22T17:18:29.552 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/122 2023-06-22T17:18:29.552 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/122 2023-06-22T17:18:29.552 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/122 2023-06-22T17:18:29.552 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/122 2023-06-22T17:18:29.552 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/122 2023-06-22T17:18:29.553 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/122 2023-06-22T17:18:29.553 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/122 2023-06-22T17:18:29.553 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/122 2023-06-22T17:18:29.553 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/122 2023-06-22T17:18:29.553 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/122 2023-06-22T17:18:29.553 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/122 2023-06-22T17:18:29.553 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/122 2023-06-22T17:18:29.553 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/122 2023-06-22T17:18:29.554 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/122 2023-06-22T17:18:29.554 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/122 2023-06-22T17:18:29.554 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/122 2023-06-22T17:18:29.554 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/122 2023-06-22T17:18:29.554 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/122 2023-06-22T17:18:29.554 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/122 2023-06-22T17:18:29.554 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/122 2023-06-22T17:18:29.554 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/122 2023-06-22T17:18:29.555 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/122 2023-06-22T17:18:29.555 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/122 2023-06-22T17:18:29.555 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/122 2023-06-22T17:18:29.555 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/122 2023-06-22T17:18:29.555 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/122 2023-06-22T17:18:29.555 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/122 2023-06-22T17:18:29.555 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 72/122 2023-06-22T17:18:29.555 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 73/122 2023-06-22T17:18:29.556 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 74/122 2023-06-22T17:18:29.556 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : jq-1.5-12.el8.x86_64 75/122 2023-06-22T17:18:29.556 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : flex-2.6.1-9.el8.x86_64 76/122 2023-06-22T17:18:29.556 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 77/122 2023-06-22T17:18:29.556 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 78/122 2023-06-22T17:18:29.556 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 79/122 2023-06-22T17:18:29.556 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : bison-3.0.4-10.el8.x86_64 80/122 2023-06-22T17:18:29.557 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 81/122 2023-06-22T17:18:29.557 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 82/122 2023-06-22T17:18:29.557 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 83/122 2023-06-22T17:18:29.557 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 84/122 2023-06-22T17:18:29.557 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 85/122 2023-06-22T17:18:29.557 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 86/122 2023-06-22T17:18:29.557 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 87/122 2023-06-22T17:18:29.557 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 88/122 2023-06-22T17:18:29.558 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 89/122 2023-06-22T17:18:29.558 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 90/122 2023-06-22T17:18:29.558 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 91/122 2023-06-22T17:18:29.558 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 92/122 2023-06-22T17:18:29.558 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 93/122 2023-06-22T17:18:29.558 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 94/122 2023-06-22T17:18:29.558 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 95/122 2023-06-22T17:18:29.558 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : automake-1.16.1-7.el8.noarch 96/122 2023-06-22T17:18:29.559 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 97/122 2023-06-22T17:18:29.559 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 98/122 2023-06-22T17:18:29.559 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 99/122 2023-06-22T17:18:29.559 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 100/122 2023-06-22T17:18:29.559 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 101/122 2023-06-22T17:18:29.559 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 102/122 2023-06-22T17:18:29.559 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 103/122 2023-06-22T17:18:29.559 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : openssl-devel-1:1.1.1g-15.el8_3.x86_64 104/122 2023-06-22T17:18:29.560 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 105/122 2023-06-22T17:18:29.560 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 106/122 2023-06-22T17:18:29.560 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 107/122 2023-06-22T17:18:29.560 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : elfutils-libelf-devel-0.182-3.el8.x86_64 108/122 2023-06-22T17:18:29.560 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 109/122 2023-06-22T17:18:29.560 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 110/122 2023-06-22T17:18:29.560 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libsepol-devel-2.9-2.el8.x86_64 111/122 2023-06-22T17:18:29.561 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 112/122 2023-06-22T17:18:29.561 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 113/122 2023-06-22T17:18:29.561 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : krb5-devel-1.18.2-8.3.el8_4.x86_64 114/122 2023-06-22T17:18:29.561 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 115/122 2023-06-22T17:18:29.561 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : xfsprogs-devel-5.0.0-8.el8.x86_64 116/122 2023-06-22T17:18:29.561 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 117/122 2023-06-22T17:18:29.561 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libuuid-devel-2.32.1-27.el8.x86_64 118/122 2023-06-22T17:18:29.562 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 119/122 2023-06-22T17:18:29.562 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 120/122 2023-06-22T17:18:29.562 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 121/122 2023-06-22T17:18:29.562 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 104/122 2023-06-22T17:18:29.607 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libtool-2.4.6-25.el8.x86_64 112/122 2023-06-22T17:18:29.618 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 112/122 2023-06-22T17:18:29.651 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 104/122 2023-06-22T17:18:29.795 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 113/122 2023-06-22T17:18:29.801 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 105/122 2023-06-22T17:18:29.845 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 105/122 2023-06-22T17:18:29.953 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : elfutils-libelf-devel-0.182-3.el8.x86_64 114/122 2023-06-22T17:18:30.010 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 106/122 2023-06-22T17:18:30.099 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 115/122 2023-06-22T17:18:30.292 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 116/122 2023-06-22T17:18:30.380 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 106/122 2023-06-22T17:18:30.551 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 107/122 2023-06-22T17:18:30.562 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 117/122 2023-06-22T17:18:30.757 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : bison-3.0.4-10.el8.x86_64 118/122 2023-06-22T17:18:30.814 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 118/122 2023-06-22T17:18:30.974 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : flex-2.6.1-9.el8.x86_64 119/122 2023-06-22T17:18:30.982 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 119/122 2023-06-22T17:18:31.101 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 120/122 2023-06-22T17:18:31.101 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-06-22T17:18:31.326 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-06-22T17:18:31.326 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 122/122 2023-06-22T17:18:31.803 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 122/122 2023-06-22T17:18:31.803 INFO:teuthology.orchestra.run.smithi133.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-06-22T17:18:31.803 INFO:teuthology.orchestra.run.smithi133.stdout:All rules containing unresolvable specifiers will be skipped. 2023-06-22T17:18:31.803 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-06-22T17:18:31.854 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 122/122 2023-06-22T17:18:31.855 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:01:42 ago on Thu 22 Jun 2023 05:16:49 PM UTC. 2023-06-22T17:18:32.145 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 1/122 2023-06-22T17:18:32.145 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2/122 2023-06-22T17:18:32.145 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 3/122 2023-06-22T17:18:32.146 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 4/122 2023-06-22T17:18:32.146 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 5/122 2023-06-22T17:18:32.146 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-22T17:18:32.146 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-22T17:18:32.146 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 8/122 2023-06-22T17:18:32.146 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 9/122 2023-06-22T17:18:32.146 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-22T17:18:32.146 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 11/122 2023-06-22T17:18:32.147 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 12/122 2023-06-22T17:18:32.147 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-22T17:18:32.147 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 14/122 2023-06-22T17:18:32.147 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 15/122 2023-06-22T17:18:32.147 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 16/122 2023-06-22T17:18:32.147 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 17/122 2023-06-22T17:18:32.147 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 18/122 2023-06-22T17:18:32.147 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 19/122 2023-06-22T17:18:32.148 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 20/122 2023-06-22T17:18:32.148 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 21/122 2023-06-22T17:18:32.148 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 22/122 2023-06-22T17:18:32.148 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 23/122 2023-06-22T17:18:32.148 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 24/122 2023-06-22T17:18:32.148 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 25/122 2023-06-22T17:18:32.155 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 26/122 2023-06-22T17:18:32.155 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 27/122 2023-06-22T17:18:32.155 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 28/122 2023-06-22T17:18:32.156 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 29/122 2023-06-22T17:18:32.156 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 30/122 2023-06-22T17:18:32.156 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 31/122 2023-06-22T17:18:32.156 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 32/122 2023-06-22T17:18:32.156 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 33/122 2023-06-22T17:18:32.156 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 34/122 2023-06-22T17:18:32.156 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 35/122 2023-06-22T17:18:32.157 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 36/122 2023-06-22T17:18:32.157 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 37/122 2023-06-22T17:18:32.157 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 38/122 2023-06-22T17:18:32.157 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/122 2023-06-22T17:18:32.157 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/122 2023-06-22T17:18:32.157 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/122 2023-06-22T17:18:32.157 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/122 2023-06-22T17:18:32.158 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/122 2023-06-22T17:18:32.158 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/122 2023-06-22T17:18:32.158 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/122 2023-06-22T17:18:32.158 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/122 2023-06-22T17:18:32.158 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/122 2023-06-22T17:18:32.158 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/122 2023-06-22T17:18:32.158 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/122 2023-06-22T17:18:32.158 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/122 2023-06-22T17:18:32.159 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/122 2023-06-22T17:18:32.159 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/122 2023-06-22T17:18:32.159 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/122 2023-06-22T17:18:32.159 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/122 2023-06-22T17:18:32.159 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/122 2023-06-22T17:18:32.159 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/122 2023-06-22T17:18:32.159 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/122 2023-06-22T17:18:32.159 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/122 2023-06-22T17:18:32.160 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/122 2023-06-22T17:18:32.160 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/122 2023-06-22T17:18:32.160 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/122 2023-06-22T17:18:32.160 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/122 2023-06-22T17:18:32.160 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/122 2023-06-22T17:18:32.160 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/122 2023-06-22T17:18:32.160 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/122 2023-06-22T17:18:32.161 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/122 2023-06-22T17:18:32.161 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/122 2023-06-22T17:18:32.161 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/122 2023-06-22T17:18:32.161 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/122 2023-06-22T17:18:32.161 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/122 2023-06-22T17:18:32.161 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/122 2023-06-22T17:18:32.161 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 72/122 2023-06-22T17:18:32.161 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 73/122 2023-06-22T17:18:32.162 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 74/122 2023-06-22T17:18:32.162 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/122 2023-06-22T17:18:32.162 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 76/122 2023-06-22T17:18:32.162 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 77/122 2023-06-22T17:18:32.162 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/122 2023-06-22T17:18:32.162 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : openssl-devel-1:1.1.1g-15.el8_3.x86_64 79/122 2023-06-22T17:18:32.162 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 80/122 2023-06-22T17:18:32.163 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 81/122 2023-06-22T17:18:32.163 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 82/122 2023-06-22T17:18:32.163 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : elfutils-libelf-devel-0.182-3.el8.x86_64 83/122 2023-06-22T17:18:32.163 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 84/122 2023-06-22T17:18:32.163 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 85/122 2023-06-22T17:18:32.164 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libsepol-devel-2.9-2.el8.x86_64 86/122 2023-06-22T17:18:32.164 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 87/122 2023-06-22T17:18:32.164 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 88/122 2023-06-22T17:18:32.164 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : krb5-devel-1.18.2-8.3.el8_4.x86_64 89/122 2023-06-22T17:18:32.164 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 90/122 2023-06-22T17:18:32.164 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : xfsprogs-devel-5.0.0-8.el8.x86_64 91/122 2023-06-22T17:18:32.164 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 92/122 2023-06-22T17:18:32.165 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libuuid-devel-2.32.1-27.el8.x86_64 93/122 2023-06-22T17:18:32.165 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 94/122 2023-06-22T17:18:32.165 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 95/122 2023-06-22T17:18:32.165 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 96/122 2023-06-22T17:18:32.165 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : jq-1.5-12.el8.x86_64 97/122 2023-06-22T17:18:32.165 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : flex-2.6.1-9.el8.x86_64 98/122 2023-06-22T17:18:32.165 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 99/122 2023-06-22T17:18:32.165 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 100/122 2023-06-22T17:18:32.166 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 101/122 2023-06-22T17:18:32.166 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : bison-3.0.4-10.el8.x86_64 102/122 2023-06-22T17:18:32.166 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 103/122 2023-06-22T17:18:32.166 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 104/122 2023-06-22T17:18:32.166 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 105/122 2023-06-22T17:18:32.166 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 106/122 2023-06-22T17:18:32.166 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/122 2023-06-22T17:18:32.166 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 108/122 2023-06-22T17:18:32.167 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 109/122 2023-06-22T17:18:32.167 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 110/122 2023-06-22T17:18:32.167 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 111/122 2023-06-22T17:18:32.167 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 112/122 2023-06-22T17:18:32.167 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 113/122 2023-06-22T17:18:32.167 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 114/122 2023-06-22T17:18:32.167 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 115/122 2023-06-22T17:18:32.168 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 116/122 2023-06-22T17:18:32.168 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 117/122 2023-06-22T17:18:32.168 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : automake-1.16.1-7.el8.noarch 118/122 2023-06-22T17:18:32.168 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 119/122 2023-06-22T17:18:32.168 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 120/122 2023-06-22T17:18:32.168 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 121/122 2023-06-22T17:18:33.801 INFO:teuthology.orchestra.run.smithi101.stdout:Installed products updated. 2023-06-22T17:18:33.924 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-06-22T17:18:33.924 INFO:teuthology.orchestra.run.smithi101.stdout:Upgraded: 2023-06-22T17:18:33.924 INFO:teuthology.orchestra.run.smithi101.stdout: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:33.924 INFO:teuthology.orchestra.run.smithi101.stdout: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:33.924 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-06-22T17:18:33.924 INFO:teuthology.orchestra.run.smithi101.stdout: automake-1.16.1-7.el8.noarch 2023-06-22T17:18:33.925 INFO:teuthology.orchestra.run.smithi101.stdout: bison-3.0.4-10.el8.x86_64 2023-06-22T17:18:33.925 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:33.925 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:33.925 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:33.925 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:33.925 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T17:18:33.925 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-immutable-object-cache-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:33.926 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:33.926 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:33.926 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T17:18:33.926 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T17:18:33.926 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-diskprediction-local-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T17:18:33.926 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T17:18:33.926 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T17:18:33.926 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:33.927 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:33.927 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T17:18:33.927 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:33.927 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:33.927 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:33.927 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T17:18:33.927 INFO:teuthology.orchestra.run.smithi101.stdout: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T17:18:33.928 INFO:teuthology.orchestra.run.smithi101.stdout: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:33.928 INFO:teuthology.orchestra.run.smithi101.stdout: cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T17:18:33.928 INFO:teuthology.orchestra.run.smithi101.stdout: elfutils-libelf-devel-0.182-3.el8.x86_64 2023-06-22T17:18:33.928 INFO:teuthology.orchestra.run.smithi101.stdout: flex-2.6.1-9.el8.x86_64 2023-06-22T17:18:33.928 INFO:teuthology.orchestra.run.smithi101.stdout: fuse-2.9.7-12.el8.x86_64 2023-06-22T17:18:33.928 INFO:teuthology.orchestra.run.smithi101.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-06-22T17:18:33.928 INFO:teuthology.orchestra.run.smithi101.stdout: jq-1.5-12.el8.x86_64 2023-06-22T17:18:33.928 INFO:teuthology.orchestra.run.smithi101.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2023-06-22T17:18:33.929 INFO:teuthology.orchestra.run.smithi101.stdout: krb5-devel-1.18.2-8.3.el8_4.x86_64 2023-06-22T17:18:33.929 INFO:teuthology.orchestra.run.smithi101.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-06-22T17:18:33.929 INFO:teuthology.orchestra.run.smithi101.stdout: libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:33.929 INFO:teuthology.orchestra.run.smithi101.stdout: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:33.929 INFO:teuthology.orchestra.run.smithi101.stdout: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:33.929 INFO:teuthology.orchestra.run.smithi101.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2023-06-22T17:18:33.929 INFO:teuthology.orchestra.run.smithi101.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-06-22T17:18:33.930 INFO:teuthology.orchestra.run.smithi101.stdout: liboath-2.6.2-3.el8.x86_64 2023-06-22T17:18:33.930 INFO:teuthology.orchestra.run.smithi101.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-06-22T17:18:33.930 INFO:teuthology.orchestra.run.smithi101.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-06-22T17:18:33.930 INFO:teuthology.orchestra.run.smithi101.stdout: librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:33.930 INFO:teuthology.orchestra.run.smithi101.stdout: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:33.930 INFO:teuthology.orchestra.run.smithi101.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-06-22T17:18:33.931 INFO:teuthology.orchestra.run.smithi101.stdout: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:33.931 INFO:teuthology.orchestra.run.smithi101.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-06-22T17:18:33.931 INFO:teuthology.orchestra.run.smithi101.stdout: libsepol-devel-2.9-2.el8.x86_64 2023-06-22T17:18:33.931 INFO:teuthology.orchestra.run.smithi101.stdout: libtool-2.4.6-25.el8.x86_64 2023-06-22T17:18:33.931 INFO:teuthology.orchestra.run.smithi101.stdout: libunwind-1.3.1-3.el8.x86_64 2023-06-22T17:18:33.931 INFO:teuthology.orchestra.run.smithi101.stdout: libuuid-devel-2.32.1-27.el8.x86_64 2023-06-22T17:18:33.931 INFO:teuthology.orchestra.run.smithi101.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-06-22T17:18:33.932 INFO:teuthology.orchestra.run.smithi101.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-06-22T17:18:33.932 INFO:teuthology.orchestra.run.smithi101.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-06-22T17:18:33.932 INFO:teuthology.orchestra.run.smithi101.stdout: openblas-0.3.12-1.el8.x86_64 2023-06-22T17:18:33.932 INFO:teuthology.orchestra.run.smithi101.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-06-22T17:18:33.932 INFO:teuthology.orchestra.run.smithi101.stdout: openssl-devel-1:1.1.1g-15.el8_3.x86_64 2023-06-22T17:18:33.932 INFO:teuthology.orchestra.run.smithi101.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-06-22T17:18:33.932 INFO:teuthology.orchestra.run.smithi101.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-06-22T17:18:33.932 INFO:teuthology.orchestra.run.smithi101.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-06-22T17:18:33.933 INFO:teuthology.orchestra.run.smithi101.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-06-22T17:18:33.933 INFO:teuthology.orchestra.run.smithi101.stdout: python3-babel-2.5.1-5.el8.noarch 2023-06-22T17:18:33.933 INFO:teuthology.orchestra.run.smithi101.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-06-22T17:18:33.933 INFO:teuthology.orchestra.run.smithi101.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-06-22T17:18:33.933 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-06-22T17:18:33.933 INFO:teuthology.orchestra.run.smithi101.stdout: python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:33.933 INFO:teuthology.orchestra.run.smithi101.stdout: python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:33.933 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:33.934 INFO:teuthology.orchestra.run.smithi101.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-06-22T17:18:33.934 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-06-22T17:18:33.934 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-06-22T17:18:33.934 INFO:teuthology.orchestra.run.smithi101.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-06-22T17:18:33.934 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jaraco-6.2-6.el8.noarch 2023-06-22T17:18:33.934 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-06-22T17:18:33.934 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-06-22T17:18:33.935 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-06-22T17:18:33.935 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-06-22T17:18:33.935 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-06-22T17:18:33.935 INFO:teuthology.orchestra.run.smithi101.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-06-22T17:18:33.935 INFO:teuthology.orchestra.run.smithi101.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-06-22T17:18:33.935 INFO:teuthology.orchestra.run.smithi101.stdout: python3-mako-1.0.6-13.el8.noarch 2023-06-22T17:18:33.935 INFO:teuthology.orchestra.run.smithi101.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-06-22T17:18:33.935 INFO:teuthology.orchestra.run.smithi101.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-06-22T17:18:33.936 INFO:teuthology.orchestra.run.smithi101.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-06-22T17:18:33.936 INFO:teuthology.orchestra.run.smithi101.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-06-22T17:18:33.936 INFO:teuthology.orchestra.run.smithi101.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-06-22T17:18:33.936 INFO:teuthology.orchestra.run.smithi101.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-06-22T17:18:33.936 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-06-22T17:18:33.936 INFO:teuthology.orchestra.run.smithi101.stdout: python3-portend-2.6-1.el8.noarch 2023-06-22T17:18:33.936 INFO:teuthology.orchestra.run.smithi101.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-06-22T17:18:33.936 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-06-22T17:18:33.937 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-06-22T17:18:33.937 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pytz-2017.2-9.el8.noarch 2023-06-22T17:18:33.937 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:33.937 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:33.937 INFO:teuthology.orchestra.run.smithi101.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-06-22T17:18:33.937 INFO:teuthology.orchestra.run.smithi101.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-06-22T17:18:33.937 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:33.938 INFO:teuthology.orchestra.run.smithi101.stdout: python3-routes-2.4.1-12.el8.noarch 2023-06-22T17:18:33.938 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rsa-4.8-1.el8.noarch 2023-06-22T17:18:33.938 INFO:teuthology.orchestra.run.smithi101.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-06-22T17:18:33.938 INFO:teuthology.orchestra.run.smithi101.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-06-22T17:18:33.938 INFO:teuthology.orchestra.run.smithi101.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-06-22T17:18:33.938 INFO:teuthology.orchestra.run.smithi101.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-06-22T17:18:33.938 INFO:teuthology.orchestra.run.smithi101.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-06-22T17:18:33.938 INFO:teuthology.orchestra.run.smithi101.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-06-22T17:18:33.939 INFO:teuthology.orchestra.run.smithi101.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-06-22T17:18:33.939 INFO:teuthology.orchestra.run.smithi101.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-06-22T17:18:33.939 INFO:teuthology.orchestra.run.smithi101.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-06-22T17:18:33.939 INFO:teuthology.orchestra.run.smithi101.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-06-22T17:18:33.939 INFO:teuthology.orchestra.run.smithi101.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-06-22T17:18:33.939 INFO:teuthology.orchestra.run.smithi101.stdout: rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:33.939 INFO:teuthology.orchestra.run.smithi101.stdout: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:33.940 INFO:teuthology.orchestra.run.smithi101.stdout: rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:33.940 INFO:teuthology.orchestra.run.smithi101.stdout: socat-1.7.3.3-2.el8.x86_64 2023-06-22T17:18:33.940 INFO:teuthology.orchestra.run.smithi101.stdout: thrift-0.13.0-2.el8.x86_64 2023-06-22T17:18:33.940 INFO:teuthology.orchestra.run.smithi101.stdout: xfsprogs-devel-5.0.0-8.el8.x86_64 2023-06-22T17:18:33.940 INFO:teuthology.orchestra.run.smithi101.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-06-22T17:18:33.940 INFO:teuthology.orchestra.run.smithi101.stdout: zlib-devel-1.2.11-17.el8.x86_64 2023-06-22T17:18:33.941 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-06-22T17:18:33.941 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-06-22T17:18:34.116 DEBUG:teuthology.parallel:result is None 2023-06-22T17:18:34.276 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 122/122 2023-06-22T17:18:34.276 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:46 ago on Thu 22 Jun 2023 05:16:48 PM UTC. 2023-06-22T17:18:35.802 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 107/122 2023-06-22T17:18:36.079 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 108/122 2023-06-22T17:18:36.174 INFO:teuthology.orchestra.run.smithi133.stdout:Installed products updated. 2023-06-22T17:18:36.225 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libtool-2.4.6-25.el8.x86_64 109/122 2023-06-22T17:18:36.293 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-06-22T17:18:36.294 INFO:teuthology.orchestra.run.smithi133.stdout:Upgraded: 2023-06-22T17:18:36.294 INFO:teuthology.orchestra.run.smithi133.stdout: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:36.294 INFO:teuthology.orchestra.run.smithi133.stdout: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:36.294 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-06-22T17:18:36.294 INFO:teuthology.orchestra.run.smithi133.stdout: automake-1.16.1-7.el8.noarch 2023-06-22T17:18:36.294 INFO:teuthology.orchestra.run.smithi133.stdout: bison-3.0.4-10.el8.x86_64 2023-06-22T17:18:36.295 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:36.295 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:36.295 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:36.295 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:36.295 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T17:18:36.295 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-immutable-object-cache-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:36.295 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:36.295 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:36.296 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T17:18:36.296 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T17:18:36.296 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-diskprediction-local-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T17:18:36.296 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T17:18:36.296 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T17:18:36.296 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:36.296 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:36.296 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T17:18:36.297 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:36.297 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:36.297 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:36.297 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T17:18:36.297 INFO:teuthology.orchestra.run.smithi133.stdout: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T17:18:36.297 INFO:teuthology.orchestra.run.smithi133.stdout: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:36.297 INFO:teuthology.orchestra.run.smithi133.stdout: cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T17:18:36.298 INFO:teuthology.orchestra.run.smithi133.stdout: elfutils-libelf-devel-0.182-3.el8.x86_64 2023-06-22T17:18:36.298 INFO:teuthology.orchestra.run.smithi133.stdout: flex-2.6.1-9.el8.x86_64 2023-06-22T17:18:36.298 INFO:teuthology.orchestra.run.smithi133.stdout: fuse-2.9.7-12.el8.x86_64 2023-06-22T17:18:36.298 INFO:teuthology.orchestra.run.smithi133.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-06-22T17:18:36.298 INFO:teuthology.orchestra.run.smithi133.stdout: jq-1.5-12.el8.x86_64 2023-06-22T17:18:36.298 INFO:teuthology.orchestra.run.smithi133.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2023-06-22T17:18:36.298 INFO:teuthology.orchestra.run.smithi133.stdout: krb5-devel-1.18.2-8.3.el8_4.x86_64 2023-06-22T17:18:36.299 INFO:teuthology.orchestra.run.smithi133.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-06-22T17:18:36.299 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:36.299 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:36.299 INFO:teuthology.orchestra.run.smithi133.stdout: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:36.299 INFO:teuthology.orchestra.run.smithi133.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2023-06-22T17:18:36.299 INFO:teuthology.orchestra.run.smithi133.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-06-22T17:18:36.299 INFO:teuthology.orchestra.run.smithi133.stdout: liboath-2.6.2-3.el8.x86_64 2023-06-22T17:18:36.299 INFO:teuthology.orchestra.run.smithi133.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-06-22T17:18:36.300 INFO:teuthology.orchestra.run.smithi133.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-06-22T17:18:36.300 INFO:teuthology.orchestra.run.smithi133.stdout: librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:36.300 INFO:teuthology.orchestra.run.smithi133.stdout: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:36.300 INFO:teuthology.orchestra.run.smithi133.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-06-22T17:18:36.300 INFO:teuthology.orchestra.run.smithi133.stdout: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:36.300 INFO:teuthology.orchestra.run.smithi133.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-06-22T17:18:36.300 INFO:teuthology.orchestra.run.smithi133.stdout: libsepol-devel-2.9-2.el8.x86_64 2023-06-22T17:18:36.300 INFO:teuthology.orchestra.run.smithi133.stdout: libtool-2.4.6-25.el8.x86_64 2023-06-22T17:18:36.301 INFO:teuthology.orchestra.run.smithi133.stdout: libunwind-1.3.1-3.el8.x86_64 2023-06-22T17:18:36.301 INFO:teuthology.orchestra.run.smithi133.stdout: libuuid-devel-2.32.1-27.el8.x86_64 2023-06-22T17:18:36.301 INFO:teuthology.orchestra.run.smithi133.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-06-22T17:18:36.301 INFO:teuthology.orchestra.run.smithi133.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-06-22T17:18:36.301 INFO:teuthology.orchestra.run.smithi133.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-06-22T17:18:36.301 INFO:teuthology.orchestra.run.smithi133.stdout: openblas-0.3.12-1.el8.x86_64 2023-06-22T17:18:36.301 INFO:teuthology.orchestra.run.smithi133.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-06-22T17:18:36.301 INFO:teuthology.orchestra.run.smithi133.stdout: openssl-devel-1:1.1.1g-15.el8_3.x86_64 2023-06-22T17:18:36.302 INFO:teuthology.orchestra.run.smithi133.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-06-22T17:18:36.302 INFO:teuthology.orchestra.run.smithi133.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-06-22T17:18:36.302 INFO:teuthology.orchestra.run.smithi133.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-06-22T17:18:36.302 INFO:teuthology.orchestra.run.smithi133.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-06-22T17:18:36.302 INFO:teuthology.orchestra.run.smithi133.stdout: python3-babel-2.5.1-5.el8.noarch 2023-06-22T17:18:36.302 INFO:teuthology.orchestra.run.smithi133.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-06-22T17:18:36.302 INFO:teuthology.orchestra.run.smithi133.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-06-22T17:18:36.302 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-06-22T17:18:36.303 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:36.303 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:36.303 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:36.303 INFO:teuthology.orchestra.run.smithi133.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-06-22T17:18:36.303 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-06-22T17:18:36.303 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-06-22T17:18:36.303 INFO:teuthology.orchestra.run.smithi133.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-06-22T17:18:36.304 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-6.2-6.el8.noarch 2023-06-22T17:18:36.304 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-06-22T17:18:36.304 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-06-22T17:18:36.304 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-06-22T17:18:36.304 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-06-22T17:18:36.304 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-06-22T17:18:36.304 INFO:teuthology.orchestra.run.smithi133.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-06-22T17:18:36.304 INFO:teuthology.orchestra.run.smithi133.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-06-22T17:18:36.305 INFO:teuthology.orchestra.run.smithi133.stdout: python3-mako-1.0.6-13.el8.noarch 2023-06-22T17:18:36.305 INFO:teuthology.orchestra.run.smithi133.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-06-22T17:18:36.305 INFO:teuthology.orchestra.run.smithi133.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-06-22T17:18:36.305 INFO:teuthology.orchestra.run.smithi133.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-06-22T17:18:36.305 INFO:teuthology.orchestra.run.smithi133.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-06-22T17:18:36.305 INFO:teuthology.orchestra.run.smithi133.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-06-22T17:18:36.305 INFO:teuthology.orchestra.run.smithi133.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-06-22T17:18:36.305 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-06-22T17:18:36.306 INFO:teuthology.orchestra.run.smithi133.stdout: python3-portend-2.6-1.el8.noarch 2023-06-22T17:18:36.306 INFO:teuthology.orchestra.run.smithi133.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-06-22T17:18:36.306 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-06-22T17:18:36.306 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-06-22T17:18:36.306 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pytz-2017.2-9.el8.noarch 2023-06-22T17:18:36.306 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:36.306 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:36.306 INFO:teuthology.orchestra.run.smithi133.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-06-22T17:18:36.307 INFO:teuthology.orchestra.run.smithi133.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-06-22T17:18:36.307 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:36.307 INFO:teuthology.orchestra.run.smithi133.stdout: python3-routes-2.4.1-12.el8.noarch 2023-06-22T17:18:36.307 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rsa-4.8-1.el8.noarch 2023-06-22T17:18:36.307 INFO:teuthology.orchestra.run.smithi133.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-06-22T17:18:36.307 INFO:teuthology.orchestra.run.smithi133.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-06-22T17:18:36.308 INFO:teuthology.orchestra.run.smithi133.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-06-22T17:18:36.308 INFO:teuthology.orchestra.run.smithi133.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-06-22T17:18:36.308 INFO:teuthology.orchestra.run.smithi133.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-06-22T17:18:36.308 INFO:teuthology.orchestra.run.smithi133.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-06-22T17:18:36.308 INFO:teuthology.orchestra.run.smithi133.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-06-22T17:18:36.308 INFO:teuthology.orchestra.run.smithi133.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-06-22T17:18:36.308 INFO:teuthology.orchestra.run.smithi133.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-06-22T17:18:36.308 INFO:teuthology.orchestra.run.smithi133.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-06-22T17:18:36.309 INFO:teuthology.orchestra.run.smithi133.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-06-22T17:18:36.309 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:36.309 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:36.309 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:36.309 INFO:teuthology.orchestra.run.smithi133.stdout: socat-1.7.3.3-2.el8.x86_64 2023-06-22T17:18:36.309 INFO:teuthology.orchestra.run.smithi133.stdout: thrift-0.13.0-2.el8.x86_64 2023-06-22T17:18:36.309 INFO:teuthology.orchestra.run.smithi133.stdout: xfsprogs-devel-5.0.0-8.el8.x86_64 2023-06-22T17:18:36.310 INFO:teuthology.orchestra.run.smithi133.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-06-22T17:18:36.310 INFO:teuthology.orchestra.run.smithi133.stdout: zlib-devel-1.2.11-17.el8.x86_64 2023-06-22T17:18:36.310 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-06-22T17:18:36.310 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-06-22T17:18:36.469 DEBUG:teuthology.parallel:result is None 2023-06-22T17:18:36.982 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 109/122 2023-06-22T17:18:37.384 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openssl-devel-1:1.1.1g-15.el8_3.x86_64 110/122 2023-06-22T17:18:37.546 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 111/122 2023-06-22T17:18:37.687 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : xfsprogs-devel-5.0.0-8.el8.x86_64 112/122 2023-06-22T17:18:37.830 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 113/122 2023-06-22T17:18:38.022 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : elfutils-libelf-devel-0.182-3.el8.x86_64 114/122 2023-06-22T17:18:38.175 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 115/122 2023-06-22T17:18:38.342 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 116/122 2023-06-22T17:18:38.476 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 117/122 2023-06-22T17:18:38.686 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 118/122 2023-06-22T17:18:38.874 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : bison-3.0.4-10.el8.x86_64 119/122 2023-06-22T17:18:38.934 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 119/122 2023-06-22T17:18:39.099 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : flex-2.6.1-9.el8.x86_64 120/122 2023-06-22T17:18:39.109 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 120/122 2023-06-22T17:18:39.109 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-06-22T17:18:39.311 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-06-22T17:18:39.311 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 122/122 2023-06-22T17:18:39.858 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 122/122 2023-06-22T17:18:39.858 INFO:teuthology.orchestra.run.smithi039.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-06-22T17:18:39.858 INFO:teuthology.orchestra.run.smithi039.stdout:All rules containing unresolvable specifiers will be skipped. 2023-06-22T17:18:39.858 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:18:40.188 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 1/122 2023-06-22T17:18:40.188 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2/122 2023-06-22T17:18:40.189 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 3/122 2023-06-22T17:18:40.189 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 4/122 2023-06-22T17:18:40.189 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 5/122 2023-06-22T17:18:40.189 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-22T17:18:40.189 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-22T17:18:40.189 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 8/122 2023-06-22T17:18:40.189 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 9/122 2023-06-22T17:18:40.189 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-22T17:18:40.190 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 11/122 2023-06-22T17:18:40.190 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 12/122 2023-06-22T17:18:40.190 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-22T17:18:40.190 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 14/122 2023-06-22T17:18:40.190 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 15/122 2023-06-22T17:18:40.190 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 16/122 2023-06-22T17:18:40.190 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 17/122 2023-06-22T17:18:40.191 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 18/122 2023-06-22T17:18:40.191 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 19/122 2023-06-22T17:18:40.191 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 20/122 2023-06-22T17:18:40.191 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 21/122 2023-06-22T17:18:40.191 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 22/122 2023-06-22T17:18:40.191 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 23/122 2023-06-22T17:18:40.191 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 24/122 2023-06-22T17:18:40.191 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 25/122 2023-06-22T17:18:40.198 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 26/122 2023-06-22T17:18:40.198 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 27/122 2023-06-22T17:18:40.198 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 28/122 2023-06-22T17:18:40.198 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 29/122 2023-06-22T17:18:40.198 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 30/122 2023-06-22T17:18:40.198 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 31/122 2023-06-22T17:18:40.198 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 32/122 2023-06-22T17:18:40.199 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 33/122 2023-06-22T17:18:40.199 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 34/122 2023-06-22T17:18:40.199 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 35/122 2023-06-22T17:18:40.199 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 36/122 2023-06-22T17:18:40.199 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 37/122 2023-06-22T17:18:40.199 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 38/122 2023-06-22T17:18:40.199 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/122 2023-06-22T17:18:40.199 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/122 2023-06-22T17:18:40.200 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/122 2023-06-22T17:18:40.200 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/122 2023-06-22T17:18:40.200 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/122 2023-06-22T17:18:40.200 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/122 2023-06-22T17:18:40.200 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/122 2023-06-22T17:18:40.200 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/122 2023-06-22T17:18:40.200 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/122 2023-06-22T17:18:40.201 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/122 2023-06-22T17:18:40.201 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/122 2023-06-22T17:18:40.201 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/122 2023-06-22T17:18:40.201 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/122 2023-06-22T17:18:40.201 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/122 2023-06-22T17:18:40.201 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/122 2023-06-22T17:18:40.201 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/122 2023-06-22T17:18:40.201 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/122 2023-06-22T17:18:40.202 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/122 2023-06-22T17:18:40.202 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/122 2023-06-22T17:18:40.202 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/122 2023-06-22T17:18:40.202 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/122 2023-06-22T17:18:40.202 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/122 2023-06-22T17:18:40.202 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/122 2023-06-22T17:18:40.202 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/122 2023-06-22T17:18:40.202 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/122 2023-06-22T17:18:40.203 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/122 2023-06-22T17:18:40.203 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/122 2023-06-22T17:18:40.203 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/122 2023-06-22T17:18:40.203 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/122 2023-06-22T17:18:40.203 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/122 2023-06-22T17:18:40.203 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/122 2023-06-22T17:18:40.203 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/122 2023-06-22T17:18:40.203 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/122 2023-06-22T17:18:40.204 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 72/122 2023-06-22T17:18:40.204 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 73/122 2023-06-22T17:18:40.204 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 74/122 2023-06-22T17:18:40.204 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : jq-1.5-12.el8.x86_64 75/122 2023-06-22T17:18:40.204 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : flex-2.6.1-9.el8.x86_64 76/122 2023-06-22T17:18:40.204 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 77/122 2023-06-22T17:18:40.204 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 78/122 2023-06-22T17:18:40.205 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 79/122 2023-06-22T17:18:40.205 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : bison-3.0.4-10.el8.x86_64 80/122 2023-06-22T17:18:40.205 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 81/122 2023-06-22T17:18:40.205 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 82/122 2023-06-22T17:18:40.205 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 83/122 2023-06-22T17:18:40.205 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 84/122 2023-06-22T17:18:40.205 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 85/122 2023-06-22T17:18:40.205 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 86/122 2023-06-22T17:18:40.206 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 87/122 2023-06-22T17:18:40.206 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 88/122 2023-06-22T17:18:40.206 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 89/122 2023-06-22T17:18:40.206 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 90/122 2023-06-22T17:18:40.206 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 91/122 2023-06-22T17:18:40.206 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 92/122 2023-06-22T17:18:40.206 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 93/122 2023-06-22T17:18:40.206 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 94/122 2023-06-22T17:18:40.207 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 95/122 2023-06-22T17:18:40.207 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : automake-1.16.1-7.el8.noarch 96/122 2023-06-22T17:18:40.207 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 97/122 2023-06-22T17:18:40.207 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 98/122 2023-06-22T17:18:40.207 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 99/122 2023-06-22T17:18:40.207 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 100/122 2023-06-22T17:18:40.207 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 101/122 2023-06-22T17:18:40.207 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 102/122 2023-06-22T17:18:40.208 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 103/122 2023-06-22T17:18:40.208 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openssl-devel-1:1.1.1g-15.el8_3.x86_64 104/122 2023-06-22T17:18:40.208 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 105/122 2023-06-22T17:18:40.208 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 106/122 2023-06-22T17:18:40.208 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 107/122 2023-06-22T17:18:40.208 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : elfutils-libelf-devel-0.182-3.el8.x86_64 108/122 2023-06-22T17:18:40.208 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 109/122 2023-06-22T17:18:40.208 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 110/122 2023-06-22T17:18:40.209 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libsepol-devel-2.9-2.el8.x86_64 111/122 2023-06-22T17:18:40.209 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 112/122 2023-06-22T17:18:40.209 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 113/122 2023-06-22T17:18:40.209 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : krb5-devel-1.18.2-8.3.el8_4.x86_64 114/122 2023-06-22T17:18:40.209 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 115/122 2023-06-22T17:18:40.209 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : xfsprogs-devel-5.0.0-8.el8.x86_64 116/122 2023-06-22T17:18:40.209 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 117/122 2023-06-22T17:18:40.209 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libuuid-devel-2.32.1-27.el8.x86_64 118/122 2023-06-22T17:18:40.210 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 119/122 2023-06-22T17:18:40.210 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 120/122 2023-06-22T17:18:40.210 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 121/122 2023-06-22T17:18:42.437 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 122/122 2023-06-22T17:18:42.437 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:51 ago on Thu 22 Jun 2023 05:16:51 PM UTC. 2023-06-22T17:18:44.425 INFO:teuthology.orchestra.run.smithi039.stdout:Installed products updated. 2023-06-22T17:18:44.557 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:18:44.558 INFO:teuthology.orchestra.run.smithi039.stdout:Upgraded: 2023-06-22T17:18:44.558 INFO:teuthology.orchestra.run.smithi039.stdout: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:44.558 INFO:teuthology.orchestra.run.smithi039.stdout: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:44.558 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-06-22T17:18:44.558 INFO:teuthology.orchestra.run.smithi039.stdout: automake-1.16.1-7.el8.noarch 2023-06-22T17:18:44.559 INFO:teuthology.orchestra.run.smithi039.stdout: bison-3.0.4-10.el8.x86_64 2023-06-22T17:18:44.559 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:44.559 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:44.559 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:44.559 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:44.559 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T17:18:44.559 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-immutable-object-cache-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:44.560 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:44.560 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:44.560 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T17:18:44.560 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T17:18:44.560 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-diskprediction-local-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T17:18:44.560 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T17:18:44.560 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T17:18:44.560 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:44.561 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:44.561 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T17:18:44.561 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:44.561 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:44.561 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:44.561 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T17:18:44.561 INFO:teuthology.orchestra.run.smithi039.stdout: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T17:18:44.561 INFO:teuthology.orchestra.run.smithi039.stdout: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:44.562 INFO:teuthology.orchestra.run.smithi039.stdout: cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T17:18:44.562 INFO:teuthology.orchestra.run.smithi039.stdout: elfutils-libelf-devel-0.182-3.el8.x86_64 2023-06-22T17:18:44.562 INFO:teuthology.orchestra.run.smithi039.stdout: flex-2.6.1-9.el8.x86_64 2023-06-22T17:18:44.562 INFO:teuthology.orchestra.run.smithi039.stdout: fuse-2.9.7-12.el8.x86_64 2023-06-22T17:18:44.562 INFO:teuthology.orchestra.run.smithi039.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-06-22T17:18:44.562 INFO:teuthology.orchestra.run.smithi039.stdout: jq-1.5-12.el8.x86_64 2023-06-22T17:18:44.562 INFO:teuthology.orchestra.run.smithi039.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2023-06-22T17:18:44.562 INFO:teuthology.orchestra.run.smithi039.stdout: krb5-devel-1.18.2-8.3.el8_4.x86_64 2023-06-22T17:18:44.563 INFO:teuthology.orchestra.run.smithi039.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-06-22T17:18:44.563 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:44.563 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:44.563 INFO:teuthology.orchestra.run.smithi039.stdout: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:44.563 INFO:teuthology.orchestra.run.smithi039.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2023-06-22T17:18:44.563 INFO:teuthology.orchestra.run.smithi039.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-06-22T17:18:44.563 INFO:teuthology.orchestra.run.smithi039.stdout: liboath-2.6.2-3.el8.x86_64 2023-06-22T17:18:44.564 INFO:teuthology.orchestra.run.smithi039.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-06-22T17:18:44.564 INFO:teuthology.orchestra.run.smithi039.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-06-22T17:18:44.564 INFO:teuthology.orchestra.run.smithi039.stdout: librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:44.564 INFO:teuthology.orchestra.run.smithi039.stdout: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:44.564 INFO:teuthology.orchestra.run.smithi039.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-06-22T17:18:44.564 INFO:teuthology.orchestra.run.smithi039.stdout: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:44.564 INFO:teuthology.orchestra.run.smithi039.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-06-22T17:18:44.564 INFO:teuthology.orchestra.run.smithi039.stdout: libsepol-devel-2.9-2.el8.x86_64 2023-06-22T17:18:44.565 INFO:teuthology.orchestra.run.smithi039.stdout: libtool-2.4.6-25.el8.x86_64 2023-06-22T17:18:44.565 INFO:teuthology.orchestra.run.smithi039.stdout: libunwind-1.3.1-3.el8.x86_64 2023-06-22T17:18:44.565 INFO:teuthology.orchestra.run.smithi039.stdout: libuuid-devel-2.32.1-27.el8.x86_64 2023-06-22T17:18:44.565 INFO:teuthology.orchestra.run.smithi039.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-06-22T17:18:44.565 INFO:teuthology.orchestra.run.smithi039.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-06-22T17:18:44.565 INFO:teuthology.orchestra.run.smithi039.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-06-22T17:18:44.565 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-0.3.12-1.el8.x86_64 2023-06-22T17:18:44.565 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-06-22T17:18:44.566 INFO:teuthology.orchestra.run.smithi039.stdout: openssl-devel-1:1.1.1g-15.el8_3.x86_64 2023-06-22T17:18:44.566 INFO:teuthology.orchestra.run.smithi039.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-06-22T17:18:44.566 INFO:teuthology.orchestra.run.smithi039.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-06-22T17:18:44.566 INFO:teuthology.orchestra.run.smithi039.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-06-22T17:18:44.566 INFO:teuthology.orchestra.run.smithi039.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-06-22T17:18:44.566 INFO:teuthology.orchestra.run.smithi039.stdout: python3-babel-2.5.1-5.el8.noarch 2023-06-22T17:18:44.566 INFO:teuthology.orchestra.run.smithi039.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-06-22T17:18:44.566 INFO:teuthology.orchestra.run.smithi039.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-06-22T17:18:44.567 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-06-22T17:18:44.567 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:44.567 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:44.567 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:44.567 INFO:teuthology.orchestra.run.smithi039.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-06-22T17:18:44.568 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-06-22T17:18:44.568 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-06-22T17:18:44.568 INFO:teuthology.orchestra.run.smithi039.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-06-22T17:18:44.568 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-6.2-6.el8.noarch 2023-06-22T17:18:44.568 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-06-22T17:18:44.569 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-06-22T17:18:44.569 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-06-22T17:18:44.569 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-06-22T17:18:44.569 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-06-22T17:18:44.569 INFO:teuthology.orchestra.run.smithi039.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-06-22T17:18:44.569 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-06-22T17:18:44.570 INFO:teuthology.orchestra.run.smithi039.stdout: python3-mako-1.0.6-13.el8.noarch 2023-06-22T17:18:44.570 INFO:teuthology.orchestra.run.smithi039.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-06-22T17:18:44.570 INFO:teuthology.orchestra.run.smithi039.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-06-22T17:18:44.570 INFO:teuthology.orchestra.run.smithi039.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-06-22T17:18:44.570 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-06-22T17:18:44.570 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-06-22T17:18:44.570 INFO:teuthology.orchestra.run.smithi039.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-06-22T17:18:44.570 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-06-22T17:18:44.571 INFO:teuthology.orchestra.run.smithi039.stdout: python3-portend-2.6-1.el8.noarch 2023-06-22T17:18:44.571 INFO:teuthology.orchestra.run.smithi039.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-06-22T17:18:44.571 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-06-22T17:18:44.571 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-06-22T17:18:44.571 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pytz-2017.2-9.el8.noarch 2023-06-22T17:18:44.572 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:44.572 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:44.572 INFO:teuthology.orchestra.run.smithi039.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-06-22T17:18:44.572 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-06-22T17:18:44.573 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:44.573 INFO:teuthology.orchestra.run.smithi039.stdout: python3-routes-2.4.1-12.el8.noarch 2023-06-22T17:18:44.573 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rsa-4.8-1.el8.noarch 2023-06-22T17:18:44.574 INFO:teuthology.orchestra.run.smithi039.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-06-22T17:18:44.574 INFO:teuthology.orchestra.run.smithi039.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-06-22T17:18:44.574 INFO:teuthology.orchestra.run.smithi039.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-06-22T17:18:44.574 INFO:teuthology.orchestra.run.smithi039.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-06-22T17:18:44.575 INFO:teuthology.orchestra.run.smithi039.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-06-22T17:18:44.575 INFO:teuthology.orchestra.run.smithi039.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-06-22T17:18:44.575 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-06-22T17:18:44.575 INFO:teuthology.orchestra.run.smithi039.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-06-22T17:18:44.576 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-06-22T17:18:44.576 INFO:teuthology.orchestra.run.smithi039.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-06-22T17:18:44.576 INFO:teuthology.orchestra.run.smithi039.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-06-22T17:18:44.577 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:44.577 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:44.577 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T17:18:44.578 INFO:teuthology.orchestra.run.smithi039.stdout: socat-1.7.3.3-2.el8.x86_64 2023-06-22T17:18:44.578 INFO:teuthology.orchestra.run.smithi039.stdout: thrift-0.13.0-2.el8.x86_64 2023-06-22T17:18:44.578 INFO:teuthology.orchestra.run.smithi039.stdout: xfsprogs-devel-5.0.0-8.el8.x86_64 2023-06-22T17:18:44.578 INFO:teuthology.orchestra.run.smithi039.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-06-22T17:18:44.579 INFO:teuthology.orchestra.run.smithi039.stdout: zlib-devel-1.2.11-17.el8.x86_64 2023-06-22T17:18:44.579 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:18:44.580 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-06-22T17:18:44.742 DEBUG:teuthology.parallel:result is None 2023-06-22T17:18:44.742 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-22T17:18:44.884 DEBUG:teuthology.orchestra.run.smithi039:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-06-22T17:18:44.906 INFO:teuthology.orchestra.run.smithi039.stdout:17.2.6-719.g42ddc46e.el8 2023-06-22T17:18:44.907 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-719.g42ddc46e.el8 2023-06-22T17:18:44.907 INFO:teuthology.task.install:The correct ceph version 17.2.6-719.g42ddc46e is installed. 2023-06-22T17:18:44.907 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-22T17:18:45.041 DEBUG:teuthology.orchestra.run.smithi101:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-06-22T17:18:45.067 INFO:teuthology.orchestra.run.smithi101.stdout:17.2.6-719.g42ddc46e.el8 2023-06-22T17:18:45.067 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-719.g42ddc46e.el8 2023-06-22T17:18:45.067 INFO:teuthology.task.install:The correct ceph version 17.2.6-719.g42ddc46e is installed. 2023-06-22T17:18:45.068 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-22T17:18:45.302 DEBUG:teuthology.orchestra.run.smithi133:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-06-22T17:18:45.328 INFO:teuthology.orchestra.run.smithi133.stdout:17.2.6-719.g42ddc46e.el8 2023-06-22T17:18:45.329 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-719.g42ddc46e.el8 2023-06-22T17:18:45.329 INFO:teuthology.task.install:The correct ceph version 17.2.6-719.g42ddc46e is installed. 2023-06-22T17:18:45.330 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-06-22T17:18:45.331 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-22T17:18:45.331 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-06-22T17:18:45.373 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-06-22T17:18:45.373 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-06-22T17:18:45.418 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-06-22T17:18:45.418 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-06-22T17:18:45.462 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-06-22T17:18:45.462 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-22T17:18:45.462 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/daemon-helper 2023-06-22T17:18:45.488 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-06-22T17:18:45.554 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-06-22T17:18:45.555 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/usr/bin/daemon-helper 2023-06-22T17:18:45.583 DEBUG:teuthology.orchestra.run.smithi101:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-06-22T17:18:45.649 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-06-22T17:18:45.649 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/usr/bin/daemon-helper 2023-06-22T17:18:45.677 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-06-22T17:18:45.742 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-06-22T17:18:45.743 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-22T17:18:45.743 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/adjust-ulimits 2023-06-22T17:18:45.772 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-06-22T17:18:45.837 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-06-22T17:18:45.838 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/usr/bin/adjust-ulimits 2023-06-22T17:18:45.863 DEBUG:teuthology.orchestra.run.smithi101:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-06-22T17:18:45.929 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-06-22T17:18:45.929 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/usr/bin/adjust-ulimits 2023-06-22T17:18:45.954 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-06-22T17:18:46.020 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-06-22T17:18:46.020 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-22T17:18:46.021 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/stdin-killer 2023-06-22T17:18:46.045 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-06-22T17:18:46.110 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-06-22T17:18:46.110 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/usr/bin/stdin-killer 2023-06-22T17:18:46.135 DEBUG:teuthology.orchestra.run.smithi101:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-06-22T17:18:46.200 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-06-22T17:18:46.201 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/usr/bin/stdin-killer 2023-06-22T17:18:46.227 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-06-22T17:18:46.291 INFO:teuthology.run_tasks:Running task cephadm... 2023-06-22T17:18:46.383 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'fuse default permissions': False, 'fuse set user groups': True, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2023-06-22T17:18:46.383 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-22T17:18:46.396 INFO:tasks.cephadm:Cluster fsid is d88002e2-1120-11ee-9b2b-001a4aab830c 2023-06-22T17:18:46.397 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-06-22T17:18:46.397 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.39', 'mon.b': '172.21.15.101', 'mon.c': '172.21.15.133'} 2023-06-22T17:18:46.398 INFO:tasks.cephadm:First mon is mon.a on smithi039 2023-06-22T17:18:46.398 INFO:tasks.cephadm:First mgr is x 2023-06-22T17:18:46.398 INFO:tasks.cephadm:Normalizing hostnames... 2023-06-22T17:18:46.399 DEBUG:teuthology.orchestra.run.smithi039:> sudo hostname $(hostname -s) 2023-06-22T17:18:46.426 DEBUG:teuthology.orchestra.run.smithi101:> sudo hostname $(hostname -s) 2023-06-22T17:18:46.452 DEBUG:teuthology.orchestra.run.smithi133:> sudo hostname $(hostname -s) 2023-06-22T17:18:46.478 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph-ci.git ref 42ddc46e6e6582d8e29a5a982a81886fe028be2c)... 2023-06-22T17:18:46.479 DEBUG:teuthology.orchestra.run.smithi039:> curl --silent https://raw.githubusercontent.com/ceph/ceph-ci/42ddc46e6e6582d8e29a5a982a81886fe028be2c/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-06-22T17:18:46.694 INFO:teuthology.orchestra.run.smithi039.stdout:-rw-rw-r--. 1 ubuntu ubuntu 366215 Jun 22 17:18 /home/ubuntu/cephtest/cephadm 2023-06-22T17:18:46.695 DEBUG:teuthology.orchestra.run.smithi101:> curl --silent https://raw.githubusercontent.com/ceph/ceph-ci/42ddc46e6e6582d8e29a5a982a81886fe028be2c/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-06-22T17:18:46.912 INFO:teuthology.orchestra.run.smithi101.stdout:-rw-rw-r--. 1 ubuntu ubuntu 366215 Jun 22 17:18 /home/ubuntu/cephtest/cephadm 2023-06-22T17:18:46.912 DEBUG:teuthology.orchestra.run.smithi133:> curl --silent https://raw.githubusercontent.com/ceph/ceph-ci/42ddc46e6e6582d8e29a5a982a81886fe028be2c/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-06-22T17:18:47.051 INFO:teuthology.orchestra.run.smithi133.stdout:-rw-rw-r--. 1 ubuntu ubuntu 366215 Jun 22 17:18 /home/ubuntu/cephtest/cephadm 2023-06-22T17:18:47.052 DEBUG:teuthology.orchestra.run.smithi039:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-06-22T17:18:47.069 DEBUG:teuthology.orchestra.run.smithi101:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-06-22T17:18:47.086 DEBUG:teuthology.orchestra.run.smithi133:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-06-22T17:18:47.121 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c on all hosts... 2023-06-22T17:18:47.121 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c pull 2023-06-22T17:18:47.123 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c pull 2023-06-22T17:18:47.128 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c pull 2023-06-22T17:18:52.108 INFO:teuthology.orchestra.run.smithi101.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c... 2023-06-22T17:18:52.145 INFO:teuthology.orchestra.run.smithi133.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c... 2023-06-22T17:18:52.855 INFO:teuthology.orchestra.run.smithi039.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c... 2023-06-22T17:19:27.374 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-06-22T17:19:27.374 INFO:teuthology.orchestra.run.smithi101.stdout: "ceph_version": "ceph version 17.2.6-719-g42ddc46e (42ddc46e6e6582d8e29a5a982a81886fe028be2c) quincy (stable)", 2023-06-22T17:19:27.375 INFO:teuthology.orchestra.run.smithi101.stdout: "image_id": "f3276efde470d44ef0ff8e259464d1d34a5015beb9d7c1215520bcf3b48fd710", 2023-06-22T17:19:27.375 INFO:teuthology.orchestra.run.smithi101.stdout: "repo_digests": [ 2023-06-22T17:19:27.375 INFO:teuthology.orchestra.run.smithi101.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b9658f1f8fdc87272ce6f577d5ecedf811fa77b9a03555eb6b1de17c25ce4236" 2023-06-22T17:19:27.375 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-06-22T17:19:27.375 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-06-22T17:19:27.408 INFO:teuthology.orchestra.run.smithi133.stdout:{ 2023-06-22T17:19:27.408 INFO:teuthology.orchestra.run.smithi133.stdout: "ceph_version": "ceph version 17.2.6-719-g42ddc46e (42ddc46e6e6582d8e29a5a982a81886fe028be2c) quincy (stable)", 2023-06-22T17:19:27.408 INFO:teuthology.orchestra.run.smithi133.stdout: "image_id": "f3276efde470d44ef0ff8e259464d1d34a5015beb9d7c1215520bcf3b48fd710", 2023-06-22T17:19:27.409 INFO:teuthology.orchestra.run.smithi133.stdout: "repo_digests": [ 2023-06-22T17:19:27.409 INFO:teuthology.orchestra.run.smithi133.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b9658f1f8fdc87272ce6f577d5ecedf811fa77b9a03555eb6b1de17c25ce4236" 2023-06-22T17:19:27.409 INFO:teuthology.orchestra.run.smithi133.stdout: ] 2023-06-22T17:19:27.409 INFO:teuthology.orchestra.run.smithi133.stdout:} 2023-06-22T17:19:27.927 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-06-22T17:19:27.927 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_version": "ceph version 17.2.6-719-g42ddc46e (42ddc46e6e6582d8e29a5a982a81886fe028be2c) quincy (stable)", 2023-06-22T17:19:27.927 INFO:teuthology.orchestra.run.smithi039.stdout: "image_id": "f3276efde470d44ef0ff8e259464d1d34a5015beb9d7c1215520bcf3b48fd710", 2023-06-22T17:19:27.927 INFO:teuthology.orchestra.run.smithi039.stdout: "repo_digests": [ 2023-06-22T17:19:27.928 INFO:teuthology.orchestra.run.smithi039.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b9658f1f8fdc87272ce6f577d5ecedf811fa77b9a03555eb6b1de17c25ce4236" 2023-06-22T17:19:27.928 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-06-22T17:19:27.928 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-06-22T17:19:27.947 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /etc/ceph 2023-06-22T17:19:27.978 DEBUG:teuthology.orchestra.run.smithi101:> sudo mkdir -p /etc/ceph 2023-06-22T17:19:28.007 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /etc/ceph 2023-06-22T17:19:28.037 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 777 /etc/ceph 2023-06-22T17:19:28.062 DEBUG:teuthology.orchestra.run.smithi101:> sudo chmod 777 /etc/ceph 2023-06-22T17:19:28.088 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 777 /etc/ceph 2023-06-22T17:19:28.114 INFO:tasks.cephadm:Writing seed config... 2023-06-22T17:19:28.116 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-06-22T17:19:28.116 INFO:tasks.cephadm: override: [client] debug client = 20 2023-06-22T17:19:28.116 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-06-22T17:19:28.117 INFO:tasks.cephadm: override: [client] fuse default permissions = False 2023-06-22T17:19:28.117 INFO:tasks.cephadm: override: [client] fuse set user groups = True 2023-06-22T17:19:28.117 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-06-22T17:19:28.117 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-06-22T17:19:28.117 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-06-22T17:19:28.118 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-06-22T17:19:28.118 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2023-06-22T17:19:28.118 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2023-06-22T17:19:28.118 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2023-06-22T17:19:28.118 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2023-06-22T17:19:28.119 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-06-22T17:19:28.119 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-06-22T17:19:28.119 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-06-22T17:19:28.119 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-06-22T17:19:28.119 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-06-22T17:19:28.119 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-06-22T17:19:28.120 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-06-22T17:19:28.120 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-06-22T17:19:28.120 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-06-22T17:19:28.120 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-06-22T17:19:28.121 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-06-22T17:19:28.121 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-06-22T17:19:28.121 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-06-22T17:19:28.121 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2023-06-22T17:19:28.121 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2023-06-22T17:19:28.122 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2023-06-22T17:19:28.122 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-06-22T17:19:28.122 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-06-22T17:19:28.122 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2023-06-22T17:19:28.122 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2023-06-22T17:19:28.123 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-06-22T17:19:28.123 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-06-22T17:19:28.123 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2023-06-22T17:19:28.123 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-06-22T17:19:28.123 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-06-22T17:19:28.124 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-06-22T17:19:28.124 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-06-22T17:19:28.124 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-06-22T17:19:28.124 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-06-22T17:19:28.124 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2023-06-22T17:19:28.125 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2023-06-22T17:19:28.125 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-22T17:19:28.126 DEBUG:teuthology.orchestra.run.smithi039:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-06-22T17:19:28.140 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = d88002e2-1120-11ee-9b2b-001a4aab830c [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 fuse default permissions = False fuse set user groups = True rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-06-22T17:19:28.141 DEBUG:teuthology.orchestra.run.smithi039:mon.a> sudo journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mon.a.service 2023-06-22T17:19:28.183 DEBUG:teuthology.orchestra.run.smithi039:mgr.x> sudo journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mgr.x.service 2023-06-22T17:19:28.207 INFO:journalctl@ceph.mon.a.smithi039.stdout:-- Logs begin at Thu 2023-06-22 17:07:05 UTC. -- 2023-06-22T17:19:28.207 INFO:tasks.cephadm:Bootstrapping... 2023-06-22T17:19:28.208 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c -v bootstrap --fsid d88002e2-1120-11ee-9b2b-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.39 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-06-22T17:19:28.231 INFO:journalctl@ceph.mgr.x.smithi039.stdout:-- Logs begin at Thu 2023-06-22 17:07:05 UTC. -- 2023-06-22T17:19:28.412 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-06-22T17:19:28.412 INFO:teuthology.orchestra.run.smithi039.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c', '-v', 'bootstrap', '--fsid', 'd88002e2-1120-11ee-9b2b-001a4aab830c', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.39', '--skip-admin-label'] 2023-06-22T17:19:28.555 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stdout 3.2.3 2023-06-22T17:19:28.555 INFO:teuthology.orchestra.run.smithi039.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-06-22T17:19:28.555 INFO:teuthology.orchestra.run.smithi039.stdout:Verifying podman|docker is present... 2023-06-22T17:19:28.695 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stdout 3.2.3 2023-06-22T17:19:28.695 INFO:teuthology.orchestra.run.smithi039.stdout:Verifying lvm2 is present... 2023-06-22T17:19:28.695 INFO:teuthology.orchestra.run.smithi039.stdout:Verifying time synchronization is in place... 2023-06-22T17:19:28.702 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-06-22T17:19:28.702 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-06-22T17:19:28.708 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-06-22T17:19:28.708 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout inactive 2023-06-22T17:19:28.714 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout enabled 2023-06-22T17:19:28.720 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout active 2023-06-22T17:19:28.720 INFO:teuthology.orchestra.run.smithi039.stdout:Unit chronyd.service is enabled and running 2023-06-22T17:19:28.721 INFO:teuthology.orchestra.run.smithi039.stdout:Repeating the final host check... 2023-06-22T17:19:28.861 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stdout 3.2.3 2023-06-22T17:19:28.861 INFO:teuthology.orchestra.run.smithi039.stdout:podman (/bin/podman) version 3.2.3 is present 2023-06-22T17:19:28.861 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl is present 2023-06-22T17:19:28.861 INFO:teuthology.orchestra.run.smithi039.stdout:lvcreate is present 2023-06-22T17:19:28.867 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-06-22T17:19:28.867 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-06-22T17:19:28.873 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-06-22T17:19:28.873 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout inactive 2023-06-22T17:19:28.879 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout enabled 2023-06-22T17:19:28.886 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout active 2023-06-22T17:19:28.886 INFO:teuthology.orchestra.run.smithi039.stdout:Unit chronyd.service is enabled and running 2023-06-22T17:19:28.886 INFO:teuthology.orchestra.run.smithi039.stdout:Host looks OK 2023-06-22T17:19:28.887 INFO:teuthology.orchestra.run.smithi039.stdout:Cluster fsid: d88002e2-1120-11ee-9b2b-001a4aab830c 2023-06-22T17:19:28.887 INFO:teuthology.orchestra.run.smithi039.stdout:Acquiring lock 140045724236264 on /run/cephadm/d88002e2-1120-11ee-9b2b-001a4aab830c.lock 2023-06-22T17:19:28.887 INFO:teuthology.orchestra.run.smithi039.stdout:Lock 140045724236264 acquired on /run/cephadm/d88002e2-1120-11ee-9b2b-001a4aab830c.lock 2023-06-22T17:19:28.888 INFO:teuthology.orchestra.run.smithi039.stdout:Verifying IP 172.21.15.39 port 3300 ... 2023-06-22T17:19:28.888 INFO:teuthology.orchestra.run.smithi039.stdout:Verifying IP 172.21.15.39 port 6789 ... 2023-06-22T17:19:28.889 INFO:teuthology.orchestra.run.smithi039.stdout:Base mon IP(s) is [172.21.15.39:3300, 172.21.15.39:6789], mon addrv is [v2:172.21.15.39:3300,v1:172.21.15.39:6789] 2023-06-22T17:19:28.890 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp metric 100 2023-06-22T17:19:28.891 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.39 metric 100 2023-06-22T17:19:28.893 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-06-22T17:19:28.893 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2023-06-22T17:19:28.894 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1798sec hoplimit 64 pref medium 2023-06-22T17:19:28.896 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-06-22T17:19:28.896 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-06-22T17:19:28.896 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-06-22T17:19:28.897 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout 3: ens1f0: mtu 1500 state UP qlen 1000 2023-06-22T17:19:28.897 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1370/64 scope link 2023-06-22T17:19:28.897 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-06-22T17:19:28.898 INFO:teuthology.orchestra.run.smithi039.stdout:Mon IP `172.21.15.39` is in CIDR network `172.21.0.0/20` 2023-06-22T17:19:28.899 INFO:teuthology.orchestra.run.smithi039.stdout:Mon IP `172.21.15.39` is in CIDR network `172.21.0.0/20` 2023-06-22T17:19:28.899 INFO:teuthology.orchestra.run.smithi039.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-06-22T17:19:28.899 INFO:teuthology.orchestra.run.smithi039.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-06-22T17:19:28.900 INFO:teuthology.orchestra.run.smithi039.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c... 2023-06-22T17:19:29.341 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stdout f3276efde470d44ef0ff8e259464d1d34a5015beb9d7c1215520bcf3b48fd710 2023-06-22T17:19:29.341 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c... 2023-06-22T17:19:29.342 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stderr Getting image source signatures 2023-06-22T17:19:29.342 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stderr Copying blob sha256:9d60a6994110d194fea96cb2ff67b650891faff48e39f8a6b8ad8152d347e90d 2023-06-22T17:19:29.342 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stderr Copying blob sha256:3dbfccf0662dd88e0c2efa9d59916ea5c98f7c5a05bbe18e86c2dc32f0494158 2023-06-22T17:19:29.342 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stderr Copying config sha256:f3276efde470d44ef0ff8e259464d1d34a5015beb9d7c1215520bcf3b48fd710 2023-06-22T17:19:29.342 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stderr Writing manifest to image destination 2023-06-22T17:19:29.342 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stderr Storing signatures 2023-06-22T17:19:30.282 INFO:teuthology.orchestra.run.smithi039.stdout:ceph: stdout ceph version 17.2.6-719-g42ddc46e (42ddc46e6e6582d8e29a5a982a81886fe028be2c) quincy (stable) 2023-06-22T17:19:30.282 INFO:teuthology.orchestra.run.smithi039.stdout:Ceph version: ceph version 17.2.6-719-g42ddc46e (42ddc46e6e6582d8e29a5a982a81886fe028be2c) quincy (stable) 2023-06-22T17:19:30.283 INFO:teuthology.orchestra.run.smithi039.stdout:Extracting ceph user uid/gid from container image... 2023-06-22T17:19:31.135 INFO:teuthology.orchestra.run.smithi039.stdout:stat: stdout 167 167 2023-06-22T17:19:31.136 INFO:teuthology.orchestra.run.smithi039.stdout:Creating initial keys... 2023-06-22T17:19:32.184 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph-authtool: stdout AQCjgpRkfKwhIxAA+XI9AlhAiPEaoBMrkiStzw== 2023-06-22T17:19:32.929 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph-authtool: stdout AQCkgpRkFdVDIBAAzjyKmDEGj7Zo3AhdtwyS0w== 2023-06-22T17:19:33.671 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph-authtool: stdout AQClgpRkEQ2OERAAaZUr26B0AwSisz3mmWl3jQ== 2023-06-22T17:19:33.671 INFO:teuthology.orchestra.run.smithi039.stdout:Creating initial monmap... 2023-06-22T17:19:34.416 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-06-22T17:19:34.416 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2023-06-22T17:19:34.417 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to d88002e2-1120-11ee-9b2b-001a4aab830c 2023-06-22T17:19:34.417 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-06-22T17:19:34.417 INFO:teuthology.orchestra.run.smithi039.stdout:monmaptool for a [v2:172.21.15.39:3300,v1:172.21.15.39:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-06-22T17:19:34.417 INFO:teuthology.orchestra.run.smithi039.stdout:setting min_mon_release = octopus 2023-06-22T17:19:34.417 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/monmaptool: set fsid to d88002e2-1120-11ee-9b2b-001a4aab830c 2023-06-22T17:19:34.417 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-06-22T17:19:34.417 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:19:34.418 INFO:teuthology.orchestra.run.smithi039.stdout:Creating mon... 2023-06-22T17:19:35.157 INFO:teuthology.orchestra.run.smithi039.stdout:create mon.a on 2023-06-22T17:19:35.266 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-06-22T17:19:35.366 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-06-22T17:19:35.474 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-d88002e2-1120-11ee-9b2b-001a4aab830c.target → /etc/systemd/system/ceph-d88002e2-1120-11ee-9b2b-001a4aab830c.target. 2023-06-22T17:19:35.474 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-d88002e2-1120-11ee-9b2b-001a4aab830c.target → /etc/systemd/system/ceph-d88002e2-1120-11ee-9b2b-001a4aab830c.target. 2023-06-22T17:19:35.615 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mon.a 2023-06-22T17:19:35.615 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Failed to reset failed state of unit ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mon.a.service: Unit ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mon.a.service not loaded. 2023-06-22T17:19:35.715 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-d88002e2-1120-11ee-9b2b-001a4aab830c.target.wants/ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mon.a.service → /etc/systemd/system/ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@.service. 2023-06-22T17:19:35.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:35 smithi039 systemd[1]: Starting Ceph mon.a for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:19:36.350 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-06-22T17:19:36.351 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout disabled 2023-06-22T17:19:36.358 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-06-22T17:19:36.359 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout inactive 2023-06-22T17:19:36.359 INFO:teuthology.orchestra.run.smithi039.stdout:firewalld.service is not enabled 2023-06-22T17:19:36.359 INFO:teuthology.orchestra.run.smithi039.stdout:Not possible to enable service . firewalld.service is not available 2023-06-22T17:19:36.359 INFO:teuthology.orchestra.run.smithi039.stdout:Waiting for mon to start... 2023-06-22T17:19:36.360 INFO:teuthology.orchestra.run.smithi039.stdout:Waiting for mon... 2023-06-22T17:19:36.559 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:36 smithi039 bash[83838]: ab6c133bdc15b23bb775611d6c93b600276c55b57cf64548e3733a6e1e751f3e 2023-06-22T17:19:36.559 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:36 smithi039 ceph-mon[83902]: mkfs d88002e2-1120-11ee-9b2b-001a4aab830c 2023-06-22T17:19:36.559 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:36 smithi039 ceph-mon[83902]: mon.a is new leader, mons a in quorum (ranks 0) 2023-06-22T17:19:36.559 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:36 smithi039 systemd[1]: Started Ceph mon.a for d88002e2-1120-11ee-9b2b-001a4aab830c. 2023-06-22T17:19:37.304 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout cluster: 2023-06-22T17:19:37.304 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout id: d88002e2-1120-11ee-9b2b-001a4aab830c 2023-06-22T17:19:37.304 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-06-22T17:19:37.305 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2023-06-22T17:19:37.305 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout services: 2023-06-22T17:19:37.305 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.579966s) 2023-06-22T17:19:37.305 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-06-22T17:19:37.305 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-06-22T17:19:37.305 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2023-06-22T17:19:37.305 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout data: 2023-06-22T17:19:37.306 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-06-22T17:19:37.306 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-06-22T17:19:37.306 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-06-22T17:19:37.306 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout pgs: 2023-06-22T17:19:37.306 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2023-06-22T17:19:37.306 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.812+0000 7fe931227700 1 Processor -- start 2023-06-22T17:19:37.306 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.812+0000 7fe931227700 1 -- start start 2023-06-22T17:19:37.306 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.812+0000 7fe931227700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe92c102900 0x7fe92c102d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:19:37.307 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.812+0000 7fe931227700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c103250 con 0x7fe92c102900 2023-06-22T17:19:37.307 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.812+0000 7fe92ad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe92c102900 0x7fe92c102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:19:37.307 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.812+0000 7fe92ad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe92c102900 0x7fe92c102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55128/0 (socket says 172.21.15.39:55128) 2023-06-22T17:19:37.307 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.812+0000 7fe92ad9d700 1 -- 172.21.15.39:0/2376640762 learned_addr learned my addr 172.21.15.39:0/2376640762 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:19:37.307 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.813+0000 7fe92ad9d700 1 -- 172.21.15.39:0/2376640762 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe92c103390 con 0x7fe92c102900 2023-06-22T17:19:37.307 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.813+0000 7fe92ad9d700 1 --2- 172.21.15.39:0/2376640762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe92c102900 0x7fe92c102d10 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe91c0098e0 tx=0x7fe91c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9c94bc0b5f6748d server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:19:37.307 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.813+0000 7fe929d9b700 1 -- 172.21.15.39:0/2376640762 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe91c009e80 con 0x7fe92c102900 2023-06-22T17:19:37.307 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.813+0000 7fe929d9b700 1 -- 172.21.15.39:0/2376640762 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe91c00b860 con 0x7fe92c102900 2023-06-22T17:19:37.308 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.813+0000 7fe929d9b700 1 -- 172.21.15.39:0/2376640762 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe91c00b9d0 con 0x7fe92c102900 2023-06-22T17:19:37.308 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.814+0000 7fe931227700 1 -- 172.21.15.39:0/2376640762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe92c102900 msgr2=0x7fe92c102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:19:37.308 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.814+0000 7fe931227700 1 --2- 172.21.15.39:0/2376640762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe92c102900 0x7fe92c102d10 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe91c0098e0 tx=0x7fe91c009250 comp rx=0 tx=0).stop 2023-06-22T17:19:37.308 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.814+0000 7fe931227700 1 -- 172.21.15.39:0/2376640762 shutdown_connections 2023-06-22T17:19:37.308 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.814+0000 7fe931227700 1 --2- 172.21.15.39:0/2376640762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe92c102900 0x7fe92c102d10 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:19:37.308 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.814+0000 7fe931227700 1 -- 172.21.15.39:0/2376640762 >> 172.21.15.39:0/2376640762 conn(0x7fe92c0fc1e0 msgr2=0x7fe92c0fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:19:37.308 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.814+0000 7fe931227700 1 -- 172.21.15.39:0/2376640762 shutdown_connections 2023-06-22T17:19:37.308 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.814+0000 7fe931227700 1 -- 172.21.15.39:0/2376640762 wait complete. 2023-06-22T17:19:37.309 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.814+0000 7fe931227700 1 Processor -- start 2023-06-22T17:19:37.309 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.814+0000 7fe931227700 1 -- start start 2023-06-22T17:19:37.309 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.815+0000 7fe931227700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe92c102900 0x7fe92c105350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:19:37.309 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.815+0000 7fe931227700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c105890 con 0x7fe92c102900 2023-06-22T17:19:37.309 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.815+0000 7fe92ad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe92c102900 0x7fe92c105350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:19:37.309 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.815+0000 7fe92ad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe92c102900 0x7fe92c105350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55144/0 (socket says 172.21.15.39:55144) 2023-06-22T17:19:37.309 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.815+0000 7fe92ad9d700 1 -- 172.21.15.39:0/384887809 learned_addr learned my addr 172.21.15.39:0/384887809 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:19:37.309 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.815+0000 7fe92ad9d700 1 -- 172.21.15.39:0/384887809 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe91c0094f0 con 0x7fe92c102900 2023-06-22T17:19:37.310 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.815+0000 7fe92ad9d700 1 --2- 172.21.15.39:0/384887809 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe92c102900 0x7fe92c105350 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe91c003080 tx=0x7fe91c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:19:37.310 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.815+0000 7fe9137fe700 1 -- 172.21.15.39:0/384887809 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe91c01c460 con 0x7fe92c102900 2023-06-22T17:19:37.310 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.815+0000 7fe9137fe700 1 -- 172.21.15.39:0/384887809 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe91c01c5c0 con 0x7fe92c102900 2023-06-22T17:19:37.310 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.815+0000 7fe931227700 1 -- 172.21.15.39:0/384887809 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe92c1039b0 con 0x7fe92c102900 2023-06-22T17:19:37.310 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.815+0000 7fe9137fe700 1 -- 172.21.15.39:0/384887809 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe91c01c890 con 0x7fe92c102900 2023-06-22T17:19:37.310 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.815+0000 7fe931227700 1 -- 172.21.15.39:0/384887809 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe92c103e20 con 0x7fe92c102900 2023-06-22T17:19:37.310 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.816+0000 7fe9137fe700 1 -- 172.21.15.39:0/384887809 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fe91c01a450 con 0x7fe92c102900 2023-06-22T17:19:37.311 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.816+0000 7fe9137fe700 1 -- 172.21.15.39:0/384887809 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fe91c02e790 con 0x7fe92c102900 2023-06-22T17:19:37.311 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.816+0000 7fe931227700 1 -- 172.21.15.39:0/384887809 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe92c05f060 con 0x7fe92c102900 2023-06-22T17:19:37.311 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.818+0000 7fe9137fe700 1 -- 172.21.15.39:0/384887809 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7fe91c01cd80 con 0x7fe92c102900 2023-06-22T17:19:37.311 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.891+0000 7fe931227700 1 -- 172.21.15.39:0/384887809 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fe92c109d10 con 0x7fe92c102900 2023-06-22T17:19:37.311 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.891+0000 7fe9137fe700 1 -- 172.21.15.39:0/384887809 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7fe91c01a890 con 0x7fe92c102900 2023-06-22T17:19:37.311 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.892+0000 7fe931227700 1 -- 172.21.15.39:0/384887809 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe92c102900 msgr2=0x7fe92c105350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:19:37.311 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.892+0000 7fe931227700 1 --2- 172.21.15.39:0/384887809 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe92c102900 0x7fe92c105350 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe91c003080 tx=0x7fe91c016000 comp rx=0 tx=0).stop 2023-06-22T17:19:37.311 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.892+0000 7fe931227700 1 -- 172.21.15.39:0/384887809 shutdown_connections 2023-06-22T17:19:37.312 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.892+0000 7fe931227700 1 --2- 172.21.15.39:0/384887809 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe92c102900 0x7fe92c105350 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:19:37.312 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.892+0000 7fe931227700 1 -- 172.21.15.39:0/384887809 >> 172.21.15.39:0/384887809 conn(0x7fe92c0fc1e0 msgr2=0x7fe92c100de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:19:37.312 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.892+0000 7fe931227700 1 -- 172.21.15.39:0/384887809 shutdown_connections 2023-06-22T17:19:37.312 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:36.892+0000 7fe931227700 1 -- 172.21.15.39:0/384887809 wait complete. 2023-06-22T17:19:37.312 INFO:teuthology.orchestra.run.smithi039.stdout:mon is available 2023-06-22T17:19:37.312 INFO:teuthology.orchestra.run.smithi039.stdout:Assimilating anything we can from ceph.conf... 2023-06-22T17:19:37.543 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:37 smithi039 ceph-mon[83902]: mon.a is new leader, mons a in quorum (ranks 0) 2023-06-22T17:19:37.543 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:37 smithi039 ceph-mon[83902]: monmap e1: 1 mons at {a=[v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0]} removed_ranks: {} 2023-06-22T17:19:37.543 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:37 smithi039 ceph-mon[83902]: fsmap 2023-06-22T17:19:37.543 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:37 smithi039 ceph-mon[83902]: osdmap e1: 0 total, 0 up, 0 in 2023-06-22T17:19:37.543 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:37 smithi039 ceph-mon[83902]: mgrmap e1: no daemons active 2023-06-22T17:19:37.544 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:37 smithi039 ceph-mon[83902]: from='client.? 172.21.15.39:0/384887809' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-06-22T17:19:38.245 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2023-06-22T17:19:38.245 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout [global] 2023-06-22T17:19:38.246 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout fsid = d88002e2-1120-11ee-9b2b-001a4aab830c 2023-06-22T17:19:38.246 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.39:3300,v1:172.21.15.39:6789] 2023-06-22T17:19:38.246 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-06-22T17:19:38.246 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-06-22T17:19:38.246 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-06-22T17:19:38.246 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-06-22T17:19:38.247 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2023-06-22T17:19:38.247 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout [mgr] 2023-06-22T17:19:38.247 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-06-22T17:19:38.247 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2023-06-22T17:19:38.247 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout [osd] 2023-06-22T17:19:38.247 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-06-22T17:19:38.247 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-06-22T17:19:38.248 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-06-22T17:19:38.248 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-06-22T17:19:38.248 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-06-22T17:19:38.248 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.765+0000 7fc1603e1700 1 Processor -- start 2023-06-22T17:19:38.248 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.765+0000 7fc1603e1700 1 -- start start 2023-06-22T17:19:38.248 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.765+0000 7fc1603e1700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc158101320 0x7fc158101730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:19:38.248 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.765+0000 7fc1603e1700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc158101c70 con 0x7fc158101320 2023-06-22T17:19:38.249 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.765+0000 7fc15e17d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc158101320 0x7fc158101730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:19:38.249 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.765+0000 7fc15e17d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc158101320 0x7fc158101730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55154/0 (socket says 172.21.15.39:55154) 2023-06-22T17:19:38.249 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.765+0000 7fc15e17d700 1 -- 172.21.15.39:0/559321915 learned_addr learned my addr 172.21.15.39:0/559321915 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:19:38.249 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.766+0000 7fc15e17d700 1 -- 172.21.15.39:0/559321915 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc158101db0 con 0x7fc158101320 2023-06-22T17:19:38.249 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.766+0000 7fc15e17d700 1 --2- 172.21.15.39:0/559321915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc158101320 0x7fc158101730 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fc1500098e0 tx=0x7fc150009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85656539721e037f server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:19:38.249 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.766+0000 7fc15d17b700 1 -- 172.21.15.39:0/559321915 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc150009e80 con 0x7fc158101320 2023-06-22T17:19:38.249 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.766+0000 7fc15d17b700 1 -- 172.21.15.39:0/559321915 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15000b860 con 0x7fc158101320 2023-06-22T17:19:38.249 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.766+0000 7fc15d17b700 1 -- 172.21.15.39:0/559321915 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc15000b9d0 con 0x7fc158101320 2023-06-22T17:19:38.250 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.766+0000 7fc1603e1700 1 -- 172.21.15.39:0/559321915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc158101320 msgr2=0x7fc158101730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:19:38.250 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.766+0000 7fc1603e1700 1 --2- 172.21.15.39:0/559321915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc158101320 0x7fc158101730 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fc1500098e0 tx=0x7fc150009250 comp rx=0 tx=0).stop 2023-06-22T17:19:38.250 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.767+0000 7fc1603e1700 1 -- 172.21.15.39:0/559321915 shutdown_connections 2023-06-22T17:19:38.250 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.767+0000 7fc1603e1700 1 --2- 172.21.15.39:0/559321915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc158101320 0x7fc158101730 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:19:38.250 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.767+0000 7fc1603e1700 1 -- 172.21.15.39:0/559321915 >> 172.21.15.39:0/559321915 conn(0x7fc1580fcac0 msgr2=0x7fc1580feed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:19:38.250 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.767+0000 7fc1603e1700 1 -- 172.21.15.39:0/559321915 shutdown_connections 2023-06-22T17:19:38.250 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.767+0000 7fc1603e1700 1 -- 172.21.15.39:0/559321915 wait complete. 2023-06-22T17:19:38.251 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.767+0000 7fc1603e1700 1 Processor -- start 2023-06-22T17:19:38.251 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.767+0000 7fc1603e1700 1 -- start start 2023-06-22T17:19:38.251 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.767+0000 7fc1603e1700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc158101320 0x7fc158110870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:19:38.251 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.767+0000 7fc1603e1700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc158110db0 con 0x7fc158101320 2023-06-22T17:19:38.251 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.767+0000 7fc15e17d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc158101320 0x7fc158110870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:19:38.251 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.767+0000 7fc15e17d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc158101320 0x7fc158110870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55164/0 (socket says 172.21.15.39:55164) 2023-06-22T17:19:38.251 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.767+0000 7fc15e17d700 1 -- 172.21.15.39:0/559819053 learned_addr learned my addr 172.21.15.39:0/559819053 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:19:38.251 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.768+0000 7fc15e17d700 1 -- 172.21.15.39:0/559819053 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1500094f0 con 0x7fc158101320 2023-06-22T17:19:38.252 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.768+0000 7fc15e17d700 1 --2- 172.21.15.39:0/559819053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc158101320 0x7fc158110870 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fc150003080 tx=0x7fc150016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:19:38.252 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.768+0000 7fc156ffd700 1 -- 172.21.15.39:0/559819053 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc15001c460 con 0x7fc158101320 2023-06-22T17:19:38.252 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.768+0000 7fc156ffd700 1 -- 172.21.15.39:0/559819053 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15001c5c0 con 0x7fc158101320 2023-06-22T17:19:38.252 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.768+0000 7fc1603e1700 1 -- 172.21.15.39:0/559819053 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc158110fb0 con 0x7fc158101320 2023-06-22T17:19:38.252 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.768+0000 7fc156ffd700 1 -- 172.21.15.39:0/559819053 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc15001c890 con 0x7fc158101320 2023-06-22T17:19:38.252 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.768+0000 7fc1603e1700 1 -- 172.21.15.39:0/559819053 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1581113c0 con 0x7fc158101320 2023-06-22T17:19:38.252 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.768+0000 7fc156ffd700 1 -- 172.21.15.39:0/559819053 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fc15001a450 con 0x7fc158101320 2023-06-22T17:19:38.253 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.768+0000 7fc156ffd700 1 -- 172.21.15.39:0/559819053 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fc15002e790 con 0x7fc158101320 2023-06-22T17:19:38.253 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.769+0000 7fc1603e1700 1 -- 172.21.15.39:0/559819053 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc15810a1e0 con 0x7fc158101320 2023-06-22T17:19:38.253 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.771+0000 7fc156ffd700 1 -- 172.21.15.39:0/559819053 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7fc15001cd80 con 0x7fc158101320 2023-06-22T17:19:38.253 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.842+0000 7fc1603e1700 1 -- 172.21.15.39:0/559819053 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fc158102690 con 0x7fc158101320 2023-06-22T17:19:38.253 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.845+0000 7fc156ffd700 1 -- 172.21.15.39:0/559819053 <== mon.0 v2:172.21.15.39:3300/0 7 ==== config(27 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc15001b3e0 con 0x7fc158101320 2023-06-22T17:19:38.253 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.846+0000 7fc156ffd700 1 -- 172.21.15.39:0/559819053 <== mon.0 v2:172.21.15.39:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7fc15002e950 con 0x7fc158101320 2023-06-22T17:19:38.253 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.846+0000 7fc1603e1700 1 -- 172.21.15.39:0/559819053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc158101320 msgr2=0x7fc158110870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:19:38.253 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.846+0000 7fc1603e1700 1 --2- 172.21.15.39:0/559819053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc158101320 0x7fc158110870 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fc150003080 tx=0x7fc150016000 comp rx=0 tx=0).stop 2023-06-22T17:19:38.254 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.846+0000 7fc1603e1700 1 -- 172.21.15.39:0/559819053 shutdown_connections 2023-06-22T17:19:38.254 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.846+0000 7fc1603e1700 1 --2- 172.21.15.39:0/559819053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc158101320 0x7fc158110870 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:19:38.254 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.846+0000 7fc1603e1700 1 -- 172.21.15.39:0/559819053 >> 172.21.15.39:0/559819053 conn(0x7fc1580fcac0 msgr2=0x7fc158076790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:19:38.254 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.847+0000 7fc1603e1700 1 -- 172.21.15.39:0/559819053 shutdown_connections 2023-06-22T17:19:38.254 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:37.847+0000 7fc1603e1700 1 -- 172.21.15.39:0/559819053 wait complete. 2023-06-22T17:19:38.254 INFO:teuthology.orchestra.run.smithi039.stdout:Generating new minimal ceph.conf... 2023-06-22T17:19:38.464 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:38 smithi039 ceph-mon[83902]: from='client.? 172.21.15.39:0/559819053' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-06-22T17:19:38.465 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:38 smithi039 ceph-mon[83902]: from='client.? 172.21.15.39:0/559819053' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-06-22T17:19:39.184 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.718+0000 7fd8662dc700 1 Processor -- start 2023-06-22T17:19:39.185 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.718+0000 7fd8662dc700 1 -- start start 2023-06-22T17:19:39.185 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.718+0000 7fd8662dc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd860102a10 0x7fd860102e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:19:39.185 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.718+0000 7fd8662dc700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd860103360 con 0x7fd860102a10 2023-06-22T17:19:39.185 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.718+0000 7fd85ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd860102a10 0x7fd860102e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:19:39.185 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.718+0000 7fd85ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd860102a10 0x7fd860102e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55176/0 (socket says 172.21.15.39:55176) 2023-06-22T17:19:39.185 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.718+0000 7fd85ffff700 1 -- 172.21.15.39:0/3527893903 learned_addr learned my addr 172.21.15.39:0/3527893903 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:19:39.186 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.718+0000 7fd85ffff700 1 -- 172.21.15.39:0/3527893903 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8601034a0 con 0x7fd860102a10 2023-06-22T17:19:39.186 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.718+0000 7fd85ffff700 1 --2- 172.21.15.39:0/3527893903 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd860102a10 0x7fd860102e20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd8500098e0 tx=0x7fd850009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41fe7feb4f078538 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:19:39.186 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.719+0000 7fd85effd700 1 -- 172.21.15.39:0/3527893903 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd850009e80 con 0x7fd860102a10 2023-06-22T17:19:39.186 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.719+0000 7fd85effd700 1 -- 172.21.15.39:0/3527893903 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd85000bdc0 con 0x7fd860102a10 2023-06-22T17:19:39.186 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.719+0000 7fd85effd700 1 -- 172.21.15.39:0/3527893903 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd850011e80 con 0x7fd860102a10 2023-06-22T17:19:39.186 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.719+0000 7fd8662dc700 1 -- 172.21.15.39:0/3527893903 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd860102a10 msgr2=0x7fd860102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:19:39.186 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.719+0000 7fd8662dc700 1 --2- 172.21.15.39:0/3527893903 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd860102a10 0x7fd860102e20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd8500098e0 tx=0x7fd850009250 comp rx=0 tx=0).stop 2023-06-22T17:19:39.186 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.719+0000 7fd8662dc700 1 -- 172.21.15.39:0/3527893903 shutdown_connections 2023-06-22T17:19:39.187 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.719+0000 7fd8662dc700 1 --2- 172.21.15.39:0/3527893903 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd860102a10 0x7fd860102e20 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:19:39.187 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.719+0000 7fd8662dc700 1 -- 172.21.15.39:0/3527893903 >> 172.21.15.39:0/3527893903 conn(0x7fd8600fc1e0 msgr2=0x7fd8600fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:19:39.187 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.719+0000 7fd8662dc700 1 -- 172.21.15.39:0/3527893903 shutdown_connections 2023-06-22T17:19:39.187 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.719+0000 7fd8662dc700 1 -- 172.21.15.39:0/3527893903 wait complete. 2023-06-22T17:19:39.187 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.720+0000 7fd8662dc700 1 Processor -- start 2023-06-22T17:19:39.187 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.720+0000 7fd8662dc700 1 -- start start 2023-06-22T17:19:39.187 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.720+0000 7fd8662dc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd860102a10 0x7fd860113270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:19:39.188 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.720+0000 7fd8662dc700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8601137b0 con 0x7fd860102a10 2023-06-22T17:19:39.188 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.720+0000 7fd85ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd860102a10 0x7fd860113270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:19:39.188 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.720+0000 7fd85ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd860102a10 0x7fd860113270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55180/0 (socket says 172.21.15.39:55180) 2023-06-22T17:19:39.188 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.720+0000 7fd85ffff700 1 -- 172.21.15.39:0/2117941972 learned_addr learned my addr 172.21.15.39:0/2117941972 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:19:39.188 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.720+0000 7fd85ffff700 1 -- 172.21.15.39:0/2117941972 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8500094f0 con 0x7fd860102a10 2023-06-22T17:19:39.188 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.721+0000 7fd85ffff700 1 --2- 172.21.15.39:0/2117941972 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd860102a10 0x7fd860113270 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd850003080 tx=0x7fd850009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:19:39.188 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.721+0000 7fd85cff9700 1 -- 172.21.15.39:0/2117941972 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd850010950 con 0x7fd860102a10 2023-06-22T17:19:39.191 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.721+0000 7fd85cff9700 1 -- 172.21.15.39:0/2117941972 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd85001a430 con 0x7fd860102a10 2023-06-22T17:19:39.191 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.721+0000 7fd8662dc700 1 -- 172.21.15.39:0/2117941972 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8601139b0 con 0x7fd860102a10 2023-06-22T17:19:39.191 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.721+0000 7fd85cff9700 1 -- 172.21.15.39:0/2117941972 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd850022400 con 0x7fd860102a10 2023-06-22T17:19:39.191 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.721+0000 7fd8662dc700 1 -- 172.21.15.39:0/2117941972 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd860113dc0 con 0x7fd860102a10 2023-06-22T17:19:39.191 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.721+0000 7fd85cff9700 1 -- 172.21.15.39:0/2117941972 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fd8500226c0 con 0x7fd860102a10 2023-06-22T17:19:39.192 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.721+0000 7fd85cff9700 1 -- 172.21.15.39:0/2117941972 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fd85002c7d0 con 0x7fd860102a10 2023-06-22T17:19:39.192 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.722+0000 7fd8662dc700 1 -- 172.21.15.39:0/2117941972 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd86005f060 con 0x7fd860102a10 2023-06-22T17:19:39.192 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.724+0000 7fd85cff9700 1 -- 172.21.15.39:0/2117941972 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7fd85001a980 con 0x7fd860102a10 2023-06-22T17:19:39.192 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.794+0000 7fd8662dc700 1 -- 172.21.15.39:0/2117941972 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fd8601088c0 con 0x7fd860102a10 2023-06-22T17:19:39.192 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.795+0000 7fd85cff9700 1 -- 172.21.15.39:0/2117941972 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fd85001a5a0 con 0x7fd860102a10 2023-06-22T17:19:39.192 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.795+0000 7fd8662dc700 1 -- 172.21.15.39:0/2117941972 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd860102a10 msgr2=0x7fd860113270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:19:39.192 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.795+0000 7fd8662dc700 1 --2- 172.21.15.39:0/2117941972 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd860102a10 0x7fd860113270 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd850003080 tx=0x7fd850009250 comp rx=0 tx=0).stop 2023-06-22T17:19:39.193 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.795+0000 7fd8662dc700 1 -- 172.21.15.39:0/2117941972 shutdown_connections 2023-06-22T17:19:39.193 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.795+0000 7fd8662dc700 1 --2- 172.21.15.39:0/2117941972 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd860102a10 0x7fd860113270 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:19:39.193 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.795+0000 7fd8662dc700 1 -- 172.21.15.39:0/2117941972 >> 172.21.15.39:0/2117941972 conn(0x7fd8600fc1e0 msgr2=0x7fd8600fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:19:39.193 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.795+0000 7fd8662dc700 1 -- 172.21.15.39:0/2117941972 shutdown_connections 2023-06-22T17:19:39.193 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:38.796+0000 7fd8662dc700 1 -- 172.21.15.39:0/2117941972 wait complete. 2023-06-22T17:19:39.193 INFO:teuthology.orchestra.run.smithi039.stdout:Restarting the monitor... 2023-06-22T17:19:39.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:39 smithi039 systemd[1]: Stopping Ceph mon.a for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:19:39.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:39 smithi039 ceph-mon[83902]: from='client.? 172.21.15.39:0/2117941972' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:19:39.775 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:39 smithi039 conmon[83878]: 2023-06-22T17:19:39.515+0000 7f6c4a56f700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-06-22T17:19:39.775 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:39 smithi039 conmon[83878]: 2023-06-22T17:19:39.515+0000 7f6c4a56f700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-06-22T17:19:39.775 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:39 smithi039 bash[84426]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c-mon-a 2023-06-22T17:19:40.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 systemd[1]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mon.a.service: Succeeded. 2023-06-22T17:19:40.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 systemd[1]: Stopped Ceph mon.a for d88002e2-1120-11ee-9b2b-001a4aab830c. 2023-06-22T17:19:40.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 systemd[1]: Starting Ceph mon.a for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:19:40.521 INFO:teuthology.orchestra.run.smithi039.stdout:Setting mon public_network to 172.21.0.0/20 2023-06-22T17:19:40.759 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: set uid:gid to 167:167 (ceph:ceph) 2023-06-22T17:19:40.759 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: ceph version 17.2.6-719-g42ddc46e (42ddc46e6e6582d8e29a5a982a81886fe028be2c) quincy (stable), process ceph-mon, pid 7 2023-06-22T17:19:40.759 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: pidfile_write: ignore empty --pid-file 2023-06-22T17:19:40.759 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: load: jerasure load: lrc 2023-06-22T17:19:40.760 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: RocksDB version: 6.15.5 2023-06-22T17:19:40.760 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-06-22T17:19:40.760 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Compile date Jun 21 2023 2023-06-22T17:19:40.760 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: DB SUMMARY 2023-06-22T17:19:40.760 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: DB Session ID: BTZDAQ9IHN0XDMGPTQKC 2023-06-22T17:19:40.760 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: CURRENT file: CURRENT 2023-06-22T17:19:40.761 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: IDENTITY file: IDENTITY 2023-06-22T17:19:40.761 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2023-06-22T17:19:40.761 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2023-06-22T17:19:40.761 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 98152 ; 2023-06-22T17:19:40.761 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.error_if_exists: 0 2023-06-22T17:19:40.761 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.create_if_missing: 0 2023-06-22T17:19:40.761 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.paranoid_checks: 1 2023-06-22T17:19:40.762 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-06-22T17:19:40.762 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.env: 0x556575d89a80 2023-06-22T17:19:40.762 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.fs: Posix File System 2023-06-22T17:19:40.762 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.info_log: 0x55657805dda0 2023-06-22T17:19:40.762 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.max_file_opening_threads: 16 2023-06-22T17:19:40.762 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.statistics: (nil) 2023-06-22T17:19:40.762 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.use_fsync: 0 2023-06-22T17:19:40.763 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.max_log_file_size: 0 2023-06-22T17:19:40.763 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-06-22T17:19:40.763 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.log_file_time_to_roll: 0 2023-06-22T17:19:40.763 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.keep_log_file_num: 1000 2023-06-22T17:19:40.763 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.recycle_log_file_num: 0 2023-06-22T17:19:40.764 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.allow_fallocate: 1 2023-06-22T17:19:40.764 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.allow_mmap_reads: 0 2023-06-22T17:19:40.764 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.allow_mmap_writes: 0 2023-06-22T17:19:40.764 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.use_direct_reads: 0 2023-06-22T17:19:40.764 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-06-22T17:19:40.764 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.create_missing_column_families: 0 2023-06-22T17:19:40.765 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.db_log_dir: 2023-06-22T17:19:40.765 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2023-06-22T17:19:40.765 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.table_cache_numshardbits: 6 2023-06-22T17:19:40.765 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.WAL_ttl_seconds: 0 2023-06-22T17:19:40.765 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.WAL_size_limit_MB: 0 2023-06-22T17:19:40.766 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-06-22T17:19:40.766 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-06-22T17:19:40.766 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.is_fd_close_on_exec: 1 2023-06-22T17:19:40.766 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.advise_random_on_open: 1 2023-06-22T17:19:40.766 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.db_write_buffer_size: 0 2023-06-22T17:19:40.766 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.write_buffer_manager: 0x556578122930 2023-06-22T17:19:40.766 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-06-22T17:19:40.767 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-06-22T17:19:40.767 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-06-22T17:19:40.767 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.use_adaptive_mutex: 0 2023-06-22T17:19:40.767 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.rate_limiter: (nil) 2023-06-22T17:19:40.767 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-06-22T17:19:40.767 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.wal_recovery_mode: 2 2023-06-22T17:19:40.767 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.enable_thread_tracking: 0 2023-06-22T17:19:40.768 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.enable_pipelined_write: 0 2023-06-22T17:19:40.768 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.unordered_write: 0 2023-06-22T17:19:40.768 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-06-22T17:19:40.768 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-06-22T17:19:40.768 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-06-22T17:19:40.768 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-06-22T17:19:40.768 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.row_cache: None 2023-06-22T17:19:40.769 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.wal_filter: None 2023-06-22T17:19:40.769 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-06-22T17:19:40.769 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.allow_ingest_behind: 0 2023-06-22T17:19:40.769 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.preserve_deletes: 0 2023-06-22T17:19:40.769 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.two_write_queues: 0 2023-06-22T17:19:40.769 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.manual_wal_flush: 0 2023-06-22T17:19:40.769 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.atomic_flush: 0 2023-06-22T17:19:40.770 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-06-22T17:19:40.770 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.persist_stats_to_disk: 0 2023-06-22T17:19:40.770 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.write_dbid_to_manifest: 0 2023-06-22T17:19:40.770 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.log_readahead_size: 0 2023-06-22T17:19:40.770 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-06-22T17:19:40.770 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.best_efforts_recovery: 0 2023-06-22T17:19:40.770 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-06-22T17:19:40.771 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-06-22T17:19:40.771 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.allow_data_in_errors: 0 2023-06-22T17:19:40.771 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.db_host_id: __hostname__ 2023-06-22T17:19:40.771 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.max_background_jobs: 2 2023-06-22T17:19:40.771 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.max_background_compactions: -1 2023-06-22T17:19:40.771 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.max_subcompactions: 1 2023-06-22T17:19:40.771 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-06-22T17:19:40.771 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-06-22T17:19:40.772 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.delayed_write_rate : 16777216 2023-06-22T17:19:40.772 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.max_total_wal_size: 0 2023-06-22T17:19:40.772 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-06-22T17:19:40.772 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.stats_dump_period_sec: 600 2023-06-22T17:19:40.772 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.stats_persist_period_sec: 600 2023-06-22T17:19:40.772 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-06-22T17:19:40.773 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.max_open_files: -1 2023-06-22T17:19:40.773 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.bytes_per_sync: 0 2023-06-22T17:19:40.773 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.wal_bytes_per_sync: 0 2023-06-22T17:19:40.773 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.strict_bytes_per_sync: 0 2023-06-22T17:19:40.773 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.compaction_readahead_size: 0 2023-06-22T17:19:40.773 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.max_background_flushes: -1 2023-06-22T17:19:40.774 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Compression algorithms supported: 2023-06-22T17:19:40.774 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-06-22T17:19:40.774 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: kZSTD supported: 0 2023-06-22T17:19:40.774 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: kXpressCompression supported: 0 2023-06-22T17:19:40.774 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: kLZ4HCCompression supported: 1 2023-06-22T17:19:40.774 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: kZlibCompression supported: 1 2023-06-22T17:19:40.774 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: kSnappyCompression supported: 1 2023-06-22T17:19:40.775 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: kLZ4Compression supported: 1 2023-06-22T17:19:40.775 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: kBZip2Compression supported: 0 2023-06-22T17:19:40.775 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Fast CRC32 supported: Supported on x86 2023-06-22T17:19:40.775 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2023-06-22T17:19:40.775 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2023-06-22T17:19:40.775 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-06-22T17:19:40.775 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.merge_operator: 2023-06-22T17:19:40.776 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.compaction_filter: None 2023-06-22T17:19:40.776 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.compaction_filter_factory: None 2023-06-22T17:19:40.776 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.sst_partitioner_factory: None 2023-06-22T17:19:40.776 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.memtable_factory: SkipListFactory 2023-06-22T17:19:40.776 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.table_factory: BlockBasedTable 2023-06-22T17:19:40.776 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55657802cd30) 2023-06-22T17:19:40.776 INFO:journalctl@ceph.mon.a.smithi039.stdout: cache_index_and_filter_blocks: 1 2023-06-22T17:19:40.777 INFO:journalctl@ceph.mon.a.smithi039.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-06-22T17:19:40.777 INFO:journalctl@ceph.mon.a.smithi039.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-06-22T17:19:40.777 INFO:journalctl@ceph.mon.a.smithi039.stdout: pin_top_level_index_and_filter: 1 2023-06-22T17:19:40.777 INFO:journalctl@ceph.mon.a.smithi039.stdout: index_type: 0 2023-06-22T17:19:40.777 INFO:journalctl@ceph.mon.a.smithi039.stdout: data_block_index_type: 0 2023-06-22T17:19:40.777 INFO:journalctl@ceph.mon.a.smithi039.stdout: index_shortening: 1 2023-06-22T17:19:40.777 INFO:journalctl@ceph.mon.a.smithi039.stdout: data_block_hash_table_util_ratio: 0.750000 2023-06-22T17:19:40.778 INFO:journalctl@ceph.mon.a.smithi039.stdout: hash_index_allow_collision: 1 2023-06-22T17:19:40.778 INFO:journalctl@ceph.mon.a.smithi039.stdout: checksum: 1 2023-06-22T17:19:40.778 INFO:journalctl@ceph.mon.a.smithi039.stdout: no_block_cache: 0 2023-06-22T17:19:40.778 INFO:journalctl@ceph.mon.a.smithi039.stdout: block_cache: 0x5565780e51f0 2023-06-22T17:19:40.778 INFO:journalctl@ceph.mon.a.smithi039.stdout: block_cache_name: BinnedLRUCache 2023-06-22T17:19:40.778 INFO:journalctl@ceph.mon.a.smithi039.stdout: block_cache_options: 2023-06-22T17:19:40.778 INFO:journalctl@ceph.mon.a.smithi039.stdout: capacity : 536870912 2023-06-22T17:19:40.779 INFO:journalctl@ceph.mon.a.smithi039.stdout: num_shard_bits : 4 2023-06-22T17:19:40.779 INFO:journalctl@ceph.mon.a.smithi039.stdout: strict_capacity_limit : 0 2023-06-22T17:19:40.779 INFO:journalctl@ceph.mon.a.smithi039.stdout: high_pri_pool_ratio: 0.000 2023-06-22T17:19:40.779 INFO:journalctl@ceph.mon.a.smithi039.stdout: block_cache_compressed: (nil) 2023-06-22T17:19:40.779 INFO:journalctl@ceph.mon.a.smithi039.stdout: persistent_cache: (nil) 2023-06-22T17:19:40.779 INFO:journalctl@ceph.mon.a.smithi039.stdout: block_size: 4096 2023-06-22T17:19:40.779 INFO:journalctl@ceph.mon.a.smithi039.stdout: block_size_deviation: 10 2023-06-22T17:19:40.779 INFO:journalctl@ceph.mon.a.smithi039.stdout: block_restart_interval: 16 2023-06-22T17:19:40.780 INFO:journalctl@ceph.mon.a.smithi039.stdout: index_block_restart_interval: 1 2023-06-22T17:19:40.780 INFO:journalctl@ceph.mon.a.smithi039.stdout: metadata_block_size: 4096 2023-06-22T17:19:40.780 INFO:journalctl@ceph.mon.a.smithi039.stdout: partition_filters: 0 2023-06-22T17:19:40.780 INFO:journalctl@ceph.mon.a.smithi039.stdout: use_delta_encoding: 1 2023-06-22T17:19:40.780 INFO:journalctl@ceph.mon.a.smithi039.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2023-06-22T17:19:40.780 INFO:journalctl@ceph.mon.a.smithi039.stdout: whole_key_filtering: 1 2023-06-22T17:19:40.780 INFO:journalctl@ceph.mon.a.smithi039.stdout: verify_compression: 0 2023-06-22T17:19:40.781 INFO:journalctl@ceph.mon.a.smithi039.stdout: read_amp_bytes_per_bit: 0 2023-06-22T17:19:40.781 INFO:journalctl@ceph.mon.a.smithi039.stdout: format_version: 4 2023-06-22T17:19:40.781 INFO:journalctl@ceph.mon.a.smithi039.stdout: enable_index_compression: 1 2023-06-22T17:19:40.781 INFO:journalctl@ceph.mon.a.smithi039.stdout: block_align: 0 2023-06-22T17:19:40.781 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.write_buffer_size: 33554432 2023-06-22T17:19:40.781 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.max_write_buffer_number: 2 2023-06-22T17:19:40.782 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.compression: NoCompression 2023-06-22T17:19:40.782 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.bottommost_compression: Disabled 2023-06-22T17:19:40.782 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.prefix_extractor: nullptr 2023-06-22T17:19:40.782 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-06-22T17:19:40.782 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.num_levels: 7 2023-06-22T17:19:40.782 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-06-22T17:19:40.782 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-06-22T17:19:40.783 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-06-22T17:19:40.783 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-06-22T17:19:40.783 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-06-22T17:19:40.783 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-06-22T17:19:40.783 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-06-22T17:19:40.784 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-06-22T17:19:40.784 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-06-22T17:19:40.784 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-06-22T17:19:40.785 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.compression_opts.window_bits: -14 2023-06-22T17:19:40.785 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.compression_opts.level: 32767 2023-06-22T17:19:40.785 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.compression_opts.strategy: 0 2023-06-22T17:19:40.785 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-06-22T17:19:40.786 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-06-22T17:19:40.786 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-06-22T17:19:40.786 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.compression_opts.enabled: false 2023-06-22T17:19:40.786 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-06-22T17:19:40.787 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-06-22T17:19:40.787 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-06-22T17:19:40.787 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.target_file_size_base: 67108864 2023-06-22T17:19:40.788 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.target_file_size_multiplier: 1 2023-06-22T17:19:40.788 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-06-22T17:19:40.788 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-06-22T17:19:40.788 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-06-22T17:19:40.789 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-06-22T17:19:40.789 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-06-22T17:19:40.789 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-06-22T17:19:40.790 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-06-22T17:19:40.790 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-06-22T17:19:40.790 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-06-22T17:19:40.790 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-06-22T17:19:40.791 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-06-22T17:19:40.791 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-06-22T17:19:40.791 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.arena_block_size: 4194304 2023-06-22T17:19:40.791 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-06-22T17:19:40.792 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-06-22T17:19:40.792 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-06-22T17:19:40.792 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.disable_auto_compactions: 0 2023-06-22T17:19:40.793 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-06-22T17:19:40.793 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-06-22T17:19:40.793 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-06-22T17:19:40.793 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-06-22T17:19:40.794 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-06-22T17:19:40.794 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-06-22T17:19:40.794 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-06-22T17:19:40.795 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-06-22T17:19:40.795 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-06-22T17:19:40.795 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-06-22T17:19:40.795 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.table_properties_collectors: 2023-06-22T17:19:40.796 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.inplace_update_support: 0 2023-06-22T17:19:40.796 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.inplace_update_num_locks: 10000 2023-06-22T17:19:40.796 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-06-22T17:19:40.797 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-06-22T17:19:40.797 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.memtable_huge_page_size: 0 2023-06-22T17:19:40.797 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.bloom_locality: 0 2023-06-22T17:19:40.797 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.max_successive_merges: 0 2023-06-22T17:19:40.798 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.optimize_filters_for_hits: 0 2023-06-22T17:19:40.798 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.paranoid_file_checks: 0 2023-06-22T17:19:40.798 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.force_consistency_checks: 1 2023-06-22T17:19:40.798 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.report_bg_io_stats: 0 2023-06-22T17:19:40.799 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.ttl: 2592000 2023-06-22T17:19:40.799 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.periodic_compaction_seconds: 0 2023-06-22T17:19:40.799 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 bash[84741]: f2fa18a6e14a9017365b276a9f04daa6d613c9ad87d8675024768ea189f86d2a 2023-06-22T17:19:40.799 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.enable_blob_files: false 2023-06-22T17:19:40.800 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.min_blob_size: 0 2023-06-22T17:19:40.800 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.blob_file_size: 268435456 2023-06-22T17:19:40.800 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.blob_compression_type: NoCompression 2023-06-22T17:19:40.801 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.enable_blob_garbage_collection: false 2023-06-22T17:19:40.801 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-06-22T17:19:40.801 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-06-22T17:19:40.801 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2023-06-22T17:19:40.802 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2023-06-22T17:19:40.802 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: EVENT_LOG_v1 {"time_micros": 1687454380501277, "job": 1, "event": "recovery_started", "wal_files": [10]} 2023-06-22T17:19:40.802 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2023-06-22T17:19:40.803 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-06-22T17:19:40.803 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: EVENT_LOG_v1 {"time_micros": 1687454380503097, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 93367, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 91450, "index_size": 277, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13614, "raw_average_key_size": 49, "raw_value_size": 83996, "raw_average_value_size": 307, "num_data_blocks": 11, "num_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1687454380, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "7453c602-edf4-4b14-9586-48da1e310deb", "db_session_id": "BTZDAQ9IHN0XDMGPTQKC"}} 2023-06-22T17:19:40.803 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2023-06-22T17:19:40.803 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: EVENT_LOG_v1 {"time_micros": 1687454380503829, "job": 1, "event": "recovery_finished"} 2023-06-22T17:19:40.804 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-06-22T17:19:40.804 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x556578079180 2023-06-22T17:19:40.804 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: DB pointer 0x556578094000 2023-06-22T17:19:40.805 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2023-06-22T17:19:40.805 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: rocksdb: [db/db_impl/db_impl.cc:903] 2023-06-22T17:19:40.805 INFO:journalctl@ceph.mon.a.smithi039.stdout: ** DB Stats ** 2023-06-22T17:19:40.805 INFO:journalctl@ceph.mon.a.smithi039.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-06-22T17:19:40.806 INFO:journalctl@ceph.mon.a.smithi039.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-06-22T17:19:40.806 INFO:journalctl@ceph.mon.a.smithi039.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-06-22T17:19:40.806 INFO:journalctl@ceph.mon.a.smithi039.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-06-22T17:19:40.806 INFO:journalctl@ceph.mon.a.smithi039.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-06-22T17:19:40.807 INFO:journalctl@ceph.mon.a.smithi039.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-06-22T17:19:40.807 INFO:journalctl@ceph.mon.a.smithi039.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-06-22T17:19:40.807 INFO:journalctl@ceph.mon.a.smithi039.stdout: 2023-06-22T17:19:40.808 INFO:journalctl@ceph.mon.a.smithi039.stdout: ** Compaction Stats [default] ** 2023-06-22T17:19:40.808 INFO:journalctl@ceph.mon.a.smithi039.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-06-22T17:19:40.808 INFO:journalctl@ceph.mon.a.smithi039.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-06-22T17:19:40.808 INFO:journalctl@ceph.mon.a.smithi039.stdout: L0 2/0 92.94 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 58.3 0.00 0.00 1 0.002 0 0 2023-06-22T17:19:40.809 INFO:journalctl@ceph.mon.a.smithi039.stdout: Sum 2/0 92.94 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 58.3 0.00 0.00 1 0.002 0 0 2023-06-22T17:19:40.809 INFO:journalctl@ceph.mon.a.smithi039.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 58.3 0.00 0.00 1 0.002 0 0 2023-06-22T17:19:40.809 INFO:journalctl@ceph.mon.a.smithi039.stdout: 2023-06-22T17:19:40.809 INFO:journalctl@ceph.mon.a.smithi039.stdout: ** Compaction Stats [default] ** 2023-06-22T17:19:40.810 INFO:journalctl@ceph.mon.a.smithi039.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-06-22T17:19:40.810 INFO:journalctl@ceph.mon.a.smithi039.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-06-22T17:19:40.810 INFO:journalctl@ceph.mon.a.smithi039.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 58.3 0.00 0.00 1 0.002 0 0 2023-06-22T17:19:40.811 INFO:journalctl@ceph.mon.a.smithi039.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-06-22T17:19:40.811 INFO:journalctl@ceph.mon.a.smithi039.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-06-22T17:19:40.811 INFO:journalctl@ceph.mon.a.smithi039.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-06-22T17:19:40.811 INFO:journalctl@ceph.mon.a.smithi039.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-06-22T17:19:40.812 INFO:journalctl@ceph.mon.a.smithi039.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-06-22T17:19:40.812 INFO:journalctl@ceph.mon.a.smithi039.stdout: AddFile(Keys): cumulative 0, interval 0 2023-06-22T17:19:40.812 INFO:journalctl@ceph.mon.a.smithi039.stdout: Cumulative compaction: 0.00 GB write, 17.74 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-06-22T17:19:40.812 INFO:journalctl@ceph.mon.a.smithi039.stdout: Interval compaction: 0.00 GB write, 17.74 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-06-22T17:19:40.813 INFO:journalctl@ceph.mon.a.smithi039.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-06-22T17:19:40.813 INFO:journalctl@ceph.mon.a.smithi039.stdout: 2023-06-22T17:19:40.813 INFO:journalctl@ceph.mon.a.smithi039.stdout: ** File Read Latency Histogram By Level [default] ** 2023-06-22T17:19:40.814 INFO:journalctl@ceph.mon.a.smithi039.stdout: 2023-06-22T17:19:40.814 INFO:journalctl@ceph.mon.a.smithi039.stdout: ** Compaction Stats [default] ** 2023-06-22T17:19:40.814 INFO:journalctl@ceph.mon.a.smithi039.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-06-22T17:19:40.814 INFO:journalctl@ceph.mon.a.smithi039.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-06-22T17:19:40.815 INFO:journalctl@ceph.mon.a.smithi039.stdout: L0 2/0 92.94 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 58.3 0.00 0.00 1 0.002 0 0 2023-06-22T17:19:40.815 INFO:journalctl@ceph.mon.a.smithi039.stdout: Sum 2/0 92.94 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 58.3 0.00 0.00 1 0.002 0 0 2023-06-22T17:19:40.815 INFO:journalctl@ceph.mon.a.smithi039.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-06-22T17:19:40.816 INFO:journalctl@ceph.mon.a.smithi039.stdout: 2023-06-22T17:19:40.816 INFO:journalctl@ceph.mon.a.smithi039.stdout: ** Compaction Stats [default] ** 2023-06-22T17:19:40.816 INFO:journalctl@ceph.mon.a.smithi039.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-06-22T17:19:40.817 INFO:journalctl@ceph.mon.a.smithi039.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-06-22T17:19:40.817 INFO:journalctl@ceph.mon.a.smithi039.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 58.3 0.00 0.00 1 0.002 0 0 2023-06-22T17:19:40.817 INFO:journalctl@ceph.mon.a.smithi039.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-06-22T17:19:40.817 INFO:journalctl@ceph.mon.a.smithi039.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-06-22T17:19:40.817 INFO:journalctl@ceph.mon.a.smithi039.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-06-22T17:19:40.817 INFO:journalctl@ceph.mon.a.smithi039.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-06-22T17:19:40.818 INFO:journalctl@ceph.mon.a.smithi039.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-06-22T17:19:40.818 INFO:journalctl@ceph.mon.a.smithi039.stdout: AddFile(Keys): cumulative 0, interval 0 2023-06-22T17:19:40.818 INFO:journalctl@ceph.mon.a.smithi039.stdout: Cumulative compaction: 0.00 GB write, 17.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-06-22T17:19:40.818 INFO:journalctl@ceph.mon.a.smithi039.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-06-22T17:19:40.818 INFO:journalctl@ceph.mon.a.smithi039.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-06-22T17:19:40.818 INFO:journalctl@ceph.mon.a.smithi039.stdout: 2023-06-22T17:19:40.818 INFO:journalctl@ceph.mon.a.smithi039.stdout: ** File Read Latency Histogram By Level [default] ** 2023-06-22T17:19:40.819 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: starting mon.a rank 0 at public addrs [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] at bind addrs [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid d88002e2-1120-11ee-9b2b-001a4aab830c 2023-06-22T17:19:40.819 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: mon.a@-1(???) e1 preinit fsid d88002e2-1120-11ee-9b2b-001a4aab830c 2023-06-22T17:19:40.819 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: mon.a@-1(???).mds e1 new map 2023-06-22T17:19:40.819 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: mon.a@-1(???).mds e1 print_map 2023-06-22T17:19:40.819 INFO:journalctl@ceph.mon.a.smithi039.stdout: e1 2023-06-22T17:19:40.819 INFO:journalctl@ceph.mon.a.smithi039.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-06-22T17:19:40.819 INFO:journalctl@ceph.mon.a.smithi039.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-06-22T17:19:40.820 INFO:journalctl@ceph.mon.a.smithi039.stdout: legacy client fscid: -1 2023-06-22T17:19:40.820 INFO:journalctl@ceph.mon.a.smithi039.stdout: 2023-06-22T17:19:40.820 INFO:journalctl@ceph.mon.a.smithi039.stdout: No filesystems configured 2023-06-22T17:19:40.820 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-06-22T17:19:40.820 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-06-22T17:19:40.820 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-06-22T17:19:40.821 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-06-22T17:19:40.821 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 ceph-mon[84805]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-06-22T17:19:40.821 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:40 smithi039 systemd[1]: Started Ceph mon.a for d88002e2-1120-11ee-9b2b-001a4aab830c. 2023-06-22T17:19:41.465 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.011+0000 7fd7abc72700 1 Processor -- start 2023-06-22T17:19:41.466 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.011+0000 7fd7abc72700 1 -- start start 2023-06-22T17:19:41.466 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.011+0000 7fd7abc72700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7a4070dc0 0x7fd7a4105010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:19:41.466 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.011+0000 7fd7abc72700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a4105550 con 0x7fd7a4070dc0 2023-06-22T17:19:41.466 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.011+0000 7fd7a9a0e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7a4070dc0 0x7fd7a4105010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:19:41.466 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.012+0000 7fd7a9a0e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7a4070dc0 0x7fd7a4105010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55194/0 (socket says 172.21.15.39:55194) 2023-06-22T17:19:41.466 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.012+0000 7fd7a9a0e700 1 -- 172.21.15.39:0/3332970226 learned_addr learned my addr 172.21.15.39:0/3332970226 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:19:41.467 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.012+0000 7fd7a9a0e700 1 -- 172.21.15.39:0/3332970226 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a4105690 con 0x7fd7a4070dc0 2023-06-22T17:19:41.467 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.012+0000 7fd7a9a0e700 1 --2- 172.21.15.39:0/3332970226 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7a4070dc0 0x7fd7a4105010 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd7a00098e0 tx=0x7fd7a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f83c4a669b963ca server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:19:41.467 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.012+0000 7fd7a8a0c700 1 -- 172.21.15.39:0/3332970226 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd7a0009e80 con 0x7fd7a4070dc0 2023-06-22T17:19:41.467 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.012+0000 7fd7a8a0c700 1 -- 172.21.15.39:0/3332970226 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd7a000bd30 con 0x7fd7a4070dc0 2023-06-22T17:19:41.467 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.012+0000 7fd7a8a0c700 1 -- 172.21.15.39:0/3332970226 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd7a0011da0 con 0x7fd7a4070dc0 2023-06-22T17:19:41.467 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.013+0000 7fd7abc72700 1 -- 172.21.15.39:0/3332970226 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7a4070dc0 msgr2=0x7fd7a4105010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:19:41.467 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.013+0000 7fd7abc72700 1 --2- 172.21.15.39:0/3332970226 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7a4070dc0 0x7fd7a4105010 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd7a00098e0 tx=0x7fd7a0009250 comp rx=0 tx=0).stop 2023-06-22T17:19:41.467 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.013+0000 7fd7abc72700 1 -- 172.21.15.39:0/3332970226 shutdown_connections 2023-06-22T17:19:41.468 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.013+0000 7fd7abc72700 1 --2- 172.21.15.39:0/3332970226 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7a4070dc0 0x7fd7a4105010 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:19:41.468 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.013+0000 7fd7abc72700 1 -- 172.21.15.39:0/3332970226 >> 172.21.15.39:0/3332970226 conn(0x7fd7a40fca60 msgr2=0x7fd7a40fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:19:41.468 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.013+0000 7fd7abc72700 1 -- 172.21.15.39:0/3332970226 shutdown_connections 2023-06-22T17:19:41.468 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.013+0000 7fd7abc72700 1 -- 172.21.15.39:0/3332970226 wait complete. 2023-06-22T17:19:41.468 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.013+0000 7fd7abc72700 1 Processor -- start 2023-06-22T17:19:41.468 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.013+0000 7fd7abc72700 1 -- start start 2023-06-22T17:19:41.468 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.014+0000 7fd7abc72700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7a4070dc0 0x7fd7a41110c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:19:41.469 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.014+0000 7fd7abc72700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a4111600 con 0x7fd7a4070dc0 2023-06-22T17:19:41.469 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.014+0000 7fd7a9a0e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7a4070dc0 0x7fd7a41110c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:19:41.469 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.014+0000 7fd7a9a0e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7a4070dc0 0x7fd7a41110c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55200/0 (socket says 172.21.15.39:55200) 2023-06-22T17:19:41.469 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.014+0000 7fd7a9a0e700 1 -- 172.21.15.39:0/1942846072 learned_addr learned my addr 172.21.15.39:0/1942846072 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:19:41.469 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.014+0000 7fd7a9a0e700 1 -- 172.21.15.39:0/1942846072 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a00094f0 con 0x7fd7a4070dc0 2023-06-22T17:19:41.469 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.014+0000 7fd7a9a0e700 1 --2- 172.21.15.39:0/1942846072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7a4070dc0 0x7fd7a41110c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd7a000b440 tx=0x7fd7a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:19:41.469 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.014+0000 7fd79a7fc700 1 -- 172.21.15.39:0/1942846072 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd7a0010b30 con 0x7fd7a4070dc0 2023-06-22T17:19:41.469 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.014+0000 7fd79a7fc700 1 -- 172.21.15.39:0/1942846072 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd7a0010c90 con 0x7fd7a4070dc0 2023-06-22T17:19:41.470 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.014+0000 7fd7abc72700 1 -- 172.21.15.39:0/1942846072 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7a4111800 con 0x7fd7a4070dc0 2023-06-22T17:19:41.470 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.014+0000 7fd79a7fc700 1 -- 172.21.15.39:0/1942846072 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd7a0022400 con 0x7fd7a4070dc0 2023-06-22T17:19:41.470 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.014+0000 7fd7abc72700 1 -- 172.21.15.39:0/1942846072 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7a4111c10 con 0x7fd7a4070dc0 2023-06-22T17:19:41.470 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.014+0000 7fd79a7fc700 1 -- 172.21.15.39:0/1942846072 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fd7a0022670 con 0x7fd7a4070dc0 2023-06-22T17:19:41.470 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.015+0000 7fd79a7fc700 1 -- 172.21.15.39:0/1942846072 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fd7a002c8b0 con 0x7fd7a4070dc0 2023-06-22T17:19:41.470 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.015+0000 7fd7abc72700 1 -- 172.21.15.39:0/1942846072 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7a405f060 con 0x7fd7a4070dc0 2023-06-22T17:19:41.470 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.017+0000 7fd79a7fc700 1 -- 172.21.15.39:0/1942846072 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7fd7a0010e00 con 0x7fd7a4070dc0 2023-06-22T17:19:41.471 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.088+0000 7fd7abc72700 1 -- 172.21.15.39:0/1942846072 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fd7a41064e0 con 0x7fd7a4070dc0 2023-06-22T17:19:41.471 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.090+0000 7fd79a7fc700 1 -- 172.21.15.39:0/1942846072 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fd7a0022b00 con 0x7fd7a4070dc0 2023-06-22T17:19:41.471 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.091+0000 7fd7abc72700 1 -- 172.21.15.39:0/1942846072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7a4070dc0 msgr2=0x7fd7a41110c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:19:41.471 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.091+0000 7fd7abc72700 1 --2- 172.21.15.39:0/1942846072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7a4070dc0 0x7fd7a41110c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd7a000b440 tx=0x7fd7a0009250 comp rx=0 tx=0).stop 2023-06-22T17:19:41.471 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.091+0000 7fd7abc72700 1 -- 172.21.15.39:0/1942846072 shutdown_connections 2023-06-22T17:19:41.471 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.091+0000 7fd7abc72700 1 --2- 172.21.15.39:0/1942846072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7a4070dc0 0x7fd7a41110c0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:19:41.471 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.091+0000 7fd7abc72700 1 -- 172.21.15.39:0/1942846072 >> 172.21.15.39:0/1942846072 conn(0x7fd7a40fca60 msgr2=0x7fd7a40fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:19:41.472 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.091+0000 7fd7abc72700 1 -- 172.21.15.39:0/1942846072 shutdown_connections 2023-06-22T17:19:41.472 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:41.091+0000 7fd7abc72700 1 -- 172.21.15.39:0/1942846072 wait complete. 2023-06-22T17:19:41.472 INFO:teuthology.orchestra.run.smithi039.stdout:Wrote config to /etc/ceph/ceph.conf 2023-06-22T17:19:41.473 INFO:teuthology.orchestra.run.smithi039.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-06-22T17:19:41.473 INFO:teuthology.orchestra.run.smithi039.stdout:Creating mgr... 2023-06-22T17:19:41.473 INFO:teuthology.orchestra.run.smithi039.stdout:Verifying port 9283 ... 2023-06-22T17:19:41.615 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mgr.x 2023-06-22T17:19:41.615 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Failed to reset failed state of unit ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mgr.x.service: Unit ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mgr.x.service not loaded. 2023-06-22T17:19:41.710 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-d88002e2-1120-11ee-9b2b-001a4aab830c.target.wants/ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mgr.x.service → /etc/systemd/system/ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@.service. 2023-06-22T17:19:41.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:41 smithi039 ceph-mon[84805]: mon.a is new leader, mons a in quorum (ranks 0) 2023-06-22T17:19:41.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:41 smithi039 ceph-mon[84805]: monmap e1: 1 mons at {a=[v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0]} removed_ranks: {} 2023-06-22T17:19:41.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:41 smithi039 ceph-mon[84805]: fsmap 2023-06-22T17:19:41.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:41 smithi039 ceph-mon[84805]: osdmap e1: 0 total, 0 up, 0 in 2023-06-22T17:19:41.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:41 smithi039 ceph-mon[84805]: mgrmap e1: no daemons active 2023-06-22T17:19:41.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:41 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/1942846072' entity='client.admin' 2023-06-22T17:19:41.933 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:19:41 smithi039 systemd[1]: Starting Ceph mgr.x for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:19:42.345 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-06-22T17:19:42.345 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout disabled 2023-06-22T17:19:42.354 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-06-22T17:19:42.354 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout inactive 2023-06-22T17:19:42.354 INFO:teuthology.orchestra.run.smithi039.stdout:firewalld.service is not enabled 2023-06-22T17:19:42.354 INFO:teuthology.orchestra.run.smithi039.stdout:Not possible to enable service . firewalld.service is not available 2023-06-22T17:19:42.362 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-06-22T17:19:42.363 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout disabled 2023-06-22T17:19:42.371 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-06-22T17:19:42.371 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout inactive 2023-06-22T17:19:42.371 INFO:teuthology.orchestra.run.smithi039.stdout:firewalld.service is not enabled 2023-06-22T17:19:42.371 INFO:teuthology.orchestra.run.smithi039.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2023-06-22T17:19:42.372 INFO:teuthology.orchestra.run.smithi039.stdout:Waiting for mgr to start... 2023-06-22T17:19:42.372 INFO:teuthology.orchestra.run.smithi039.stdout:Waiting for mgr... 2023-06-22T17:19:42.558 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:19:42 smithi039 bash[85177]: a47e7e88ead87c01c5c10a989a0563e7a0ae56789d5c17598b5f4efdca998493 2023-06-22T17:19:42.558 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:19:42 smithi039 systemd[1]: Started Ceph mgr.x for d88002e2-1120-11ee-9b2b-001a4aab830c. 2023-06-22T17:19:43.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:42 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/248976067' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-06-22T17:19:43.320 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2023-06-22T17:19:43.320 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout { 2023-06-22T17:19:43.320 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "fsid": "d88002e2-1120-11ee-9b2b-001a4aab830c", 2023-06-22T17:19:43.320 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "health": { 2023-06-22T17:19:43.320 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-06-22T17:19:43.321 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-06-22T17:19:43.321 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-06-22T17:19:43.321 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2023-06-22T17:19:43.321 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-06-22T17:19:43.321 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-06-22T17:19:43.321 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 0 2023-06-22T17:19:43.321 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2023-06-22T17:19:43.321 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-06-22T17:19:43.322 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "a" 2023-06-22T17:19:43.322 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2023-06-22T17:19:43.322 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2023-06-22T17:19:43.322 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "monmap": { 2023-06-22T17:19:43.322 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-06-22T17:19:43.322 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2023-06-22T17:19:43.322 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-06-22T17:19:43.322 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2023-06-22T17:19:43.323 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-06-22T17:19:43.323 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-06-22T17:19:43.323 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-06-22T17:19:43.323 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-06-22T17:19:43.323 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-06-22T17:19:43.323 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-06-22T17:19:43.323 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-06-22T17:19:43.323 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-06-22T17:19:43.324 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2023-06-22T17:19:43.324 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-06-22T17:19:43.324 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-06-22T17:19:43.324 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-06-22T17:19:43.324 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-06-22T17:19:43.324 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-06-22T17:19:43.324 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-06-22T17:19:43.325 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-06-22T17:19:43.325 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-06-22T17:19:43.325 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-06-22T17:19:43.325 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2023-06-22T17:19:43.325 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-06-22T17:19:43.325 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-06-22T17:19:43.325 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-06-22T17:19:43.325 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-06-22T17:19:43.326 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2023-06-22T17:19:43.326 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-06-22T17:19:43.326 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "available": false, 2023-06-22T17:19:43.326 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-06-22T17:19:43.326 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "modules": [ 2023-06-22T17:19:43.326 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "iostat", 2023-06-22T17:19:43.326 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "nfs", 2023-06-22T17:19:43.326 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "restful" 2023-06-22T17:19:43.327 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2023-06-22T17:19:43.327 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "services": {} 2023-06-22T17:19:43.327 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2023-06-22T17:19:43.327 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-06-22T17:19:43.327 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-06-22T17:19:43.327 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "modified": "2023-06-22T17:19:36.314220+0000", 2023-06-22T17:19:43.327 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "services": {} 2023-06-22T17:19:43.327 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2023-06-22T17:19:43.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-06-22T17:19:43.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout } 2023-06-22T17:19:43.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.819+0000 7f91cbcaf700 1 Processor -- start 2023-06-22T17:19:43.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.819+0000 7f91cbcaf700 1 -- start start 2023-06-22T17:19:43.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.820+0000 7f91cbcaf700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91c4070e00 0x7f91c4104f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:19:43.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.820+0000 7f91cbcaf700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c41054b0 con 0x7f91c4070e00 2023-06-22T17:19:43.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.820+0000 7f91c9a4b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91c4070e00 0x7f91c4104f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:19:43.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.820+0000 7f91c9a4b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91c4070e00 0x7f91c4104f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55226/0 (socket says 172.21.15.39:55226) 2023-06-22T17:19:43.329 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.820+0000 7f91c9a4b700 1 -- 172.21.15.39:0/1643825963 learned_addr learned my addr 172.21.15.39:0/1643825963 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:19:43.329 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.820+0000 7f91c9a4b700 1 -- 172.21.15.39:0/1643825963 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c41055f0 con 0x7f91c4070e00 2023-06-22T17:19:43.329 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.820+0000 7f91c9a4b700 1 --2- 172.21.15.39:0/1643825963 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91c4070e00 0x7f91c4104f70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f91c00098e0 tx=0x7f91c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9bff0af49a7dae5 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:19:43.329 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.821+0000 7f91c8a49700 1 -- 172.21.15.39:0/1643825963 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f91c0009e80 con 0x7f91c4070e00 2023-06-22T17:19:43.329 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.821+0000 7f91c8a49700 1 -- 172.21.15.39:0/1643825963 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f91c000bd30 con 0x7f91c4070e00 2023-06-22T17:19:43.329 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.821+0000 7f91c8a49700 1 -- 172.21.15.39:0/1643825963 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f91c0011da0 con 0x7f91c4070e00 2023-06-22T17:19:43.329 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.821+0000 7f91cbcaf700 1 -- 172.21.15.39:0/1643825963 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91c4070e00 msgr2=0x7f91c4104f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:19:43.329 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.821+0000 7f91cbcaf700 1 --2- 172.21.15.39:0/1643825963 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91c4070e00 0x7f91c4104f70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f91c00098e0 tx=0x7f91c0009250 comp rx=0 tx=0).stop 2023-06-22T17:19:43.330 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.821+0000 7f91cbcaf700 1 -- 172.21.15.39:0/1643825963 shutdown_connections 2023-06-22T17:19:43.330 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.821+0000 7f91cbcaf700 1 --2- 172.21.15.39:0/1643825963 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91c4070e00 0x7f91c4104f70 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:19:43.330 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.821+0000 7f91cbcaf700 1 -- 172.21.15.39:0/1643825963 >> 172.21.15.39:0/1643825963 conn(0x7f91c40fc9e0 msgr2=0x7f91c40fee30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:19:43.330 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.821+0000 7f91cbcaf700 1 -- 172.21.15.39:0/1643825963 shutdown_connections 2023-06-22T17:19:43.330 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.821+0000 7f91cbcaf700 1 -- 172.21.15.39:0/1643825963 wait complete. 2023-06-22T17:19:43.331 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.822+0000 7f91cbcaf700 1 Processor -- start 2023-06-22T17:19:43.331 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.822+0000 7f91cbcaf700 1 -- start start 2023-06-22T17:19:43.331 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.822+0000 7f91cbcaf700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91c4070e00 0x7f91c4102810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:19:43.331 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.822+0000 7f91cbcaf700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c4102d50 con 0x7f91c4070e00 2023-06-22T17:19:43.331 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.822+0000 7f91c9a4b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91c4070e00 0x7f91c4102810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:19:43.331 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.822+0000 7f91c9a4b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91c4070e00 0x7f91c4102810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55234/0 (socket says 172.21.15.39:55234) 2023-06-22T17:19:43.331 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.822+0000 7f91c9a4b700 1 -- 172.21.15.39:0/248976067 learned_addr learned my addr 172.21.15.39:0/248976067 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:19:43.332 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.822+0000 7f91c9a4b700 1 -- 172.21.15.39:0/248976067 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c00094f0 con 0x7f91c4070e00 2023-06-22T17:19:43.332 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.823+0000 7f91c9a4b700 1 --2- 172.21.15.39:0/248976067 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91c4070e00 0x7f91c4102810 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f91c0006d40 tx=0x7f91c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:19:43.332 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.823+0000 7f91ba7fc700 1 -- 172.21.15.39:0/248976067 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f91c0010b60 con 0x7f91c4070e00 2023-06-22T17:19:43.332 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.823+0000 7f91ba7fc700 1 -- 172.21.15.39:0/248976067 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f91c0010cc0 con 0x7f91c4070e00 2023-06-22T17:19:43.332 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.823+0000 7f91cbcaf700 1 -- 172.21.15.39:0/248976067 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91c4100e70 con 0x7f91c4070e00 2023-06-22T17:19:43.332 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.823+0000 7f91ba7fc700 1 -- 172.21.15.39:0/248976067 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f91c0022400 con 0x7f91c4070e00 2023-06-22T17:19:43.332 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.823+0000 7f91cbcaf700 1 -- 172.21.15.39:0/248976067 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91c41012e0 con 0x7f91c4070e00 2023-06-22T17:19:43.333 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.823+0000 7f91ba7fc700 1 -- 172.21.15.39:0/248976067 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f91c0022650 con 0x7f91c4070e00 2023-06-22T17:19:43.333 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.823+0000 7f91ba7fc700 1 -- 172.21.15.39:0/248976067 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f91c002c8a0 con 0x7f91c4070e00 2023-06-22T17:19:43.333 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.824+0000 7f91cbcaf700 1 -- 172.21.15.39:0/248976067 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91c41064a0 con 0x7f91c4070e00 2023-06-22T17:19:43.333 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.826+0000 7f91ba7fc700 1 -- 172.21.15.39:0/248976067 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f91c0010e30 con 0x7f91c4070e00 2023-06-22T17:19:43.333 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.899+0000 7f91cbcaf700 1 -- 172.21.15.39:0/248976067 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f91c4101880 con 0x7f91c4070e00 2023-06-22T17:19:43.333 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.899+0000 7f91ba7fc700 1 -- 172.21.15.39:0/248976067 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (secure 0 0 0) 0x7f91c001a990 con 0x7f91c4070e00 2023-06-22T17:19:43.333 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.900+0000 7f91cbcaf700 1 -- 172.21.15.39:0/248976067 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91c4070e00 msgr2=0x7f91c4102810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:19:43.333 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.900+0000 7f91cbcaf700 1 --2- 172.21.15.39:0/248976067 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91c4070e00 0x7f91c4102810 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f91c0006d40 tx=0x7f91c0009250 comp rx=0 tx=0).stop 2023-06-22T17:19:43.334 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.900+0000 7f91cbcaf700 1 -- 172.21.15.39:0/248976067 shutdown_connections 2023-06-22T17:19:43.334 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.900+0000 7f91cbcaf700 1 --2- 172.21.15.39:0/248976067 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91c4070e00 0x7f91c4102810 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:19:43.334 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.900+0000 7f91cbcaf700 1 -- 172.21.15.39:0/248976067 >> 172.21.15.39:0/248976067 conn(0x7f91c40fc9e0 msgr2=0x7f91c40fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:19:43.334 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.900+0000 7f91cbcaf700 1 -- 172.21.15.39:0/248976067 shutdown_connections 2023-06-22T17:19:43.334 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:42.901+0000 7f91cbcaf700 1 -- 172.21.15.39:0/248976067 wait complete. 2023-06-22T17:19:43.334 INFO:teuthology.orchestra.run.smithi039.stdout:mgr not available, waiting (1/15)... 2023-06-22T17:19:43.832 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:19:43 smithi039 conmon[85217]: 2023-06-22T17:19:43.574+0000 7f3aeff031c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-06-22T17:19:43.832 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:19:43 smithi039 conmon[85217]: 2023-06-22T17:19:43.739+0000 7f3aeff031c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-06-22T17:19:44.182 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:19:43 smithi039 conmon[85217]: 2023-06-22T17:19:43.831+0000 7f3aeff031c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-06-22T17:19:44.182 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:19:44 smithi039 conmon[85217]: 2023-06-22T17:19:44.053+0000 7f3aeff031c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-06-22T17:19:44.682 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:19:44 smithi039 conmon[85217]: 2023-06-22T17:19:44.263+0000 7f3aeff031c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-06-22T17:19:44.682 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:19:44 smithi039 conmon[85217]: 2023-06-22T17:19:44.428+0000 7f3aeff031c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-06-22T17:19:44.682 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:19:44 smithi039 conmon[85217]: 2023-06-22T17:19:44.510+0000 7f3aeff031c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-06-22T17:19:45.432 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:19:45 smithi039 conmon[85217]: 2023-06-22T17:19:45.132+0000 7f3aeff031c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-06-22T17:19:45.848 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:19:45 smithi039 conmon[85217]: 2023-06-22T17:19:45.651+0000 7f3aeff031c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-06-22T17:19:45.848 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:19:45 smithi039 conmon[85217]: 2023-06-22T17:19:45.749+0000 7f3aeff031c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-06-22T17:19:45.849 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:19:45 smithi039 conmon[85217]: 2023-06-22T17:19:45.847+0000 7f3aeff031c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-06-22T17:19:46.259 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2023-06-22T17:19:46.259 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout { 2023-06-22T17:19:46.260 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "fsid": "d88002e2-1120-11ee-9b2b-001a4aab830c", 2023-06-22T17:19:46.260 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "health": { 2023-06-22T17:19:46.260 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-06-22T17:19:46.260 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-06-22T17:19:46.261 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-06-22T17:19:46.261 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2023-06-22T17:19:46.261 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-06-22T17:19:46.261 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-06-22T17:19:46.261 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 0 2023-06-22T17:19:46.262 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2023-06-22T17:19:46.262 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-06-22T17:19:46.262 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "a" 2023-06-22T17:19:46.262 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2023-06-22T17:19:46.262 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2023-06-22T17:19:46.263 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "monmap": { 2023-06-22T17:19:46.263 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-06-22T17:19:46.263 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2023-06-22T17:19:46.263 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-06-22T17:19:46.263 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2023-06-22T17:19:46.263 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-06-22T17:19:46.263 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-06-22T17:19:46.263 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-06-22T17:19:46.264 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-06-22T17:19:46.264 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-06-22T17:19:46.270 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-06-22T17:19:46.270 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-06-22T17:19:46.270 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-06-22T17:19:46.270 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2023-06-22T17:19:46.270 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-06-22T17:19:46.270 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-06-22T17:19:46.271 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-06-22T17:19:46.271 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-06-22T17:19:46.271 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-06-22T17:19:46.271 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-06-22T17:19:46.271 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-06-22T17:19:46.271 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-06-22T17:19:46.271 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-06-22T17:19:46.271 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2023-06-22T17:19:46.272 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-06-22T17:19:46.272 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-06-22T17:19:46.272 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-06-22T17:19:46.272 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-06-22T17:19:46.272 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2023-06-22T17:19:46.272 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-06-22T17:19:46.273 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "available": false, 2023-06-22T17:19:46.273 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-06-22T17:19:46.273 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "modules": [ 2023-06-22T17:19:46.273 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "iostat", 2023-06-22T17:19:46.273 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "nfs", 2023-06-22T17:19:46.274 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "restful" 2023-06-22T17:19:46.274 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2023-06-22T17:19:46.274 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "services": {} 2023-06-22T17:19:46.274 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2023-06-22T17:19:46.275 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-06-22T17:19:46.275 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-06-22T17:19:46.275 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "modified": "2023-06-22T17:19:36.314220+0000", 2023-06-22T17:19:46.275 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "services": {} 2023-06-22T17:19:46.276 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2023-06-22T17:19:46.276 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-06-22T17:19:46.276 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout } 2023-06-22T17:19:46.276 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.784+0000 7f2d75230700 1 Processor -- start 2023-06-22T17:19:46.277 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.784+0000 7f2d75230700 1 -- start start 2023-06-22T17:19:46.277 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.784+0000 7f2d75230700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d70070dc0 0x7f2d70105010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:19:46.277 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.784+0000 7f2d75230700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d70105550 con 0x7f2d70070dc0 2023-06-22T17:19:46.277 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.784+0000 7f2d6ed9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d70070dc0 0x7f2d70105010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:19:46.278 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.784+0000 7f2d6ed9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d70070dc0 0x7f2d70105010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:60460/0 (socket says 172.21.15.39:60460) 2023-06-22T17:19:46.278 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.784+0000 7f2d6ed9d700 1 -- 172.21.15.39:0/270486065 learned_addr learned my addr 172.21.15.39:0/270486065 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:19:46.278 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.784+0000 7f2d6ed9d700 1 -- 172.21.15.39:0/270486065 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d70105690 con 0x7f2d70070dc0 2023-06-22T17:19:46.278 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.785+0000 7f2d6ed9d700 1 --2- 172.21.15.39:0/270486065 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d70070dc0 0x7f2d70105010 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2d600098e0 tx=0x7f2d60009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd1b76f2d7d21e1f server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:19:46.279 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.785+0000 7f2d6dd9b700 1 -- 172.21.15.39:0/270486065 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2d60009e80 con 0x7f2d70070dc0 2023-06-22T17:19:46.279 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.785+0000 7f2d6dd9b700 1 -- 172.21.15.39:0/270486065 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2d6000bd30 con 0x7f2d70070dc0 2023-06-22T17:19:46.279 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.785+0000 7f2d6dd9b700 1 -- 172.21.15.39:0/270486065 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2d60011da0 con 0x7f2d70070dc0 2023-06-22T17:19:46.279 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.785+0000 7f2d75230700 1 -- 172.21.15.39:0/270486065 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d70070dc0 msgr2=0x7f2d70105010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:19:46.280 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.785+0000 7f2d75230700 1 --2- 172.21.15.39:0/270486065 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d70070dc0 0x7f2d70105010 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2d600098e0 tx=0x7f2d60009250 comp rx=0 tx=0).stop 2023-06-22T17:19:46.280 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.785+0000 7f2d75230700 1 -- 172.21.15.39:0/270486065 shutdown_connections 2023-06-22T17:19:46.280 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.785+0000 7f2d75230700 1 --2- 172.21.15.39:0/270486065 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d70070dc0 0x7f2d70105010 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:19:46.280 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.785+0000 7f2d75230700 1 -- 172.21.15.39:0/270486065 >> 172.21.15.39:0/270486065 conn(0x7f2d700fca60 msgr2=0x7f2d700fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:19:46.280 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.786+0000 7f2d75230700 1 -- 172.21.15.39:0/270486065 shutdown_connections 2023-06-22T17:19:46.280 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.786+0000 7f2d75230700 1 -- 172.21.15.39:0/270486065 wait complete. 2023-06-22T17:19:46.281 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.786+0000 7f2d75230700 1 Processor -- start 2023-06-22T17:19:46.281 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.786+0000 7f2d75230700 1 -- start start 2023-06-22T17:19:46.281 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.786+0000 7f2d75230700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d70070dc0 0x7f2d701110c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:19:46.281 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.786+0000 7f2d75230700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d70111600 con 0x7f2d70070dc0 2023-06-22T17:19:46.281 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.786+0000 7f2d6ed9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d70070dc0 0x7f2d701110c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:19:46.281 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.787+0000 7f2d6ed9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d70070dc0 0x7f2d701110c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:60462/0 (socket says 172.21.15.39:60462) 2023-06-22T17:19:46.281 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.787+0000 7f2d6ed9d700 1 -- 172.21.15.39:0/1535702476 learned_addr learned my addr 172.21.15.39:0/1535702476 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:19:46.282 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.787+0000 7f2d6ed9d700 1 -- 172.21.15.39:0/1535702476 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d600094f0 con 0x7f2d70070dc0 2023-06-22T17:19:46.282 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.787+0000 7f2d6ed9d700 1 --2- 172.21.15.39:0/1535702476 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d70070dc0 0x7f2d701110c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2d60006fd0 tx=0x7f2d60009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:19:46.282 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.787+0000 7f2d577fe700 1 -- 172.21.15.39:0/1535702476 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2d60010b30 con 0x7f2d70070dc0 2023-06-22T17:19:46.282 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.787+0000 7f2d75230700 1 -- 172.21.15.39:0/1535702476 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d70111800 con 0x7f2d70070dc0 2023-06-22T17:19:46.282 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.787+0000 7f2d577fe700 1 -- 172.21.15.39:0/1535702476 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2d60010c90 con 0x7f2d70070dc0 2023-06-22T17:19:46.282 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.787+0000 7f2d577fe700 1 -- 172.21.15.39:0/1535702476 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2d60022400 con 0x7f2d70070dc0 2023-06-22T17:19:46.282 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.787+0000 7f2d75230700 1 -- 172.21.15.39:0/1535702476 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d70111c10 con 0x7f2d70070dc0 2023-06-22T17:19:46.282 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.787+0000 7f2d577fe700 1 -- 172.21.15.39:0/1535702476 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f2d600226c0 con 0x7f2d70070dc0 2023-06-22T17:19:46.283 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.787+0000 7f2d577fe700 1 -- 172.21.15.39:0/1535702476 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f2d6002c910 con 0x7f2d70070dc0 2023-06-22T17:19:46.283 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.788+0000 7f2d75230700 1 -- 172.21.15.39:0/1535702476 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d7005f060 con 0x7f2d70070dc0 2023-06-22T17:19:46.283 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.792+0000 7f2d577fe700 1 -- 172.21.15.39:0/1535702476 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f2d60010e00 con 0x7f2d70070dc0 2023-06-22T17:19:46.283 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.866+0000 7f2d75230700 1 -- 172.21.15.39:0/1535702476 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f2d70106680 con 0x7f2d70070dc0 2023-06-22T17:19:46.283 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.867+0000 7f2d577fe700 1 -- 172.21.15.39:0/1535702476 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (secure 0 0 0) 0x7f2d60022b50 con 0x7f2d70070dc0 2023-06-22T17:19:46.283 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.868+0000 7f2d75230700 1 -- 172.21.15.39:0/1535702476 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d70070dc0 msgr2=0x7f2d701110c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:19:46.283 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.868+0000 7f2d75230700 1 --2- 172.21.15.39:0/1535702476 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d70070dc0 0x7f2d701110c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2d60006fd0 tx=0x7f2d60009250 comp rx=0 tx=0).stop 2023-06-22T17:19:46.283 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.868+0000 7f2d75230700 1 -- 172.21.15.39:0/1535702476 shutdown_connections 2023-06-22T17:19:46.284 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.868+0000 7f2d75230700 1 --2- 172.21.15.39:0/1535702476 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d70070dc0 0x7f2d701110c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:19:46.284 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.868+0000 7f2d75230700 1 -- 172.21.15.39:0/1535702476 >> 172.21.15.39:0/1535702476 conn(0x7f2d700fca60 msgr2=0x7f2d700fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:19:46.284 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.868+0000 7f2d75230700 1 -- 172.21.15.39:0/1535702476 shutdown_connections 2023-06-22T17:19:46.284 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:45.868+0000 7f2d75230700 1 -- 172.21.15.39:0/1535702476 wait complete. 2023-06-22T17:19:46.284 INFO:teuthology.orchestra.run.smithi039.stdout:mgr not available, waiting (2/15)... 2023-06-22T17:19:46.514 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:19:46 smithi039 conmon[85217]: 2023-06-22T17:19:46.473+0000 7f3aeff031c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-06-22T17:19:46.880 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:46 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/1535702476' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-06-22T17:19:46.880 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:19:46 smithi039 conmon[85217]: 2023-06-22T17:19:46.649+0000 7f3aeff031c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-06-22T17:19:46.880 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:19:46 smithi039 conmon[85217]: 2023-06-22T17:19:46.744+0000 7f3aeff031c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-06-22T17:19:47.182 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:19:46 smithi039 conmon[85217]: 2023-06-22T17:19:46.879+0000 7f3aeff031c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-06-22T17:19:47.182 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:19:46 smithi039 conmon[85217]: 2023-06-22T17:19:46.975+0000 7f3aeff031c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-06-22T17:19:48.182 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:19:47 smithi039 conmon[85217]: 2023-06-22T17:19:47.906+0000 7f3aeff031c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-06-22T17:19:48.182 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:19:47 smithi039 conmon[85217]: 2023-06-22T17:19:47.997+0000 7f3aeff031c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-06-22T17:19:48.182 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:19:48 smithi039 conmon[85217]: 2023-06-22T17:19:48.091+0000 7f3aeff031c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-06-22T17:19:48.682 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:19:48 smithi039 conmon[85217]: 2023-06-22T17:19:48.256+0000 7f3aeff031c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-06-22T17:19:48.682 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:19:48 smithi039 conmon[85217]: 2023-06-22T17:19:48.369+0000 7f3aeff031c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-06-22T17:19:49.052 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:19:48 smithi039 conmon[85217]: 2023-06-22T17:19:48.775+0000 7f3aeff031c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-06-22T17:19:49.053 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:19:48 smithi039 conmon[85217]: 2023-06-22T17:19:48.868+0000 7f3aeff031c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-06-22T17:19:49.053 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:19:48 smithi039 conmon[85217]: 2023-06-22T17:19:48.959+0000 7f3aeff031c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-06-22T17:19:49.432 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:19:49 smithi039 conmon[85217]: 2023-06-22T17:19:49.051+0000 7f3aeff031c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-06-22T17:19:49.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:49 smithi039 ceph-mon[84805]: Activating manager daemon x 2023-06-22T17:19:49.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:49 smithi039 ceph-mon[84805]: mgrmap e2: x(active, starting, since 0.00518327s) 2023-06-22T17:19:49.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:49 smithi039 ceph-mon[84805]: from='mgr.14100 172.21.15.39:0/4155684329' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-06-22T17:19:49.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:49 smithi039 ceph-mon[84805]: from='mgr.14100 172.21.15.39:0/4155684329' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-06-22T17:19:49.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:49 smithi039 ceph-mon[84805]: from='mgr.14100 172.21.15.39:0/4155684329' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-06-22T17:19:49.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:49 smithi039 ceph-mon[84805]: from='mgr.14100 172.21.15.39:0/4155684329' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-22T17:19:49.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:49 smithi039 ceph-mon[84805]: from='mgr.14100 172.21.15.39:0/4155684329' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-06-22T17:19:49.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:49 smithi039 ceph-mon[84805]: Manager daemon x is now available 2023-06-22T17:19:49.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:49 smithi039 ceph-mon[84805]: from='mgr.14100 172.21.15.39:0/4155684329' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-06-22T17:19:49.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:49 smithi039 ceph-mon[84805]: from='mgr.14100 172.21.15.39:0/4155684329' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-06-22T17:19:49.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:49 smithi039 ceph-mon[84805]: from='mgr.14100 172.21.15.39:0/4155684329' entity='mgr.x' 2023-06-22T17:19:49.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:49 smithi039 ceph-mon[84805]: from='mgr.14100 172.21.15.39:0/4155684329' entity='mgr.x' 2023-06-22T17:19:49.435 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:49 smithi039 ceph-mon[84805]: from='mgr.14100 172.21.15.39:0/4155684329' entity='mgr.x' 2023-06-22T17:19:51.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:51 smithi039 ceph-mon[84805]: mgrmap e3: x(active, since 1.01085s) 2023-06-22T17:19:51.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:51 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/3983313361' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-06-22T17:19:52.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:52 smithi039 ceph-mon[84805]: mgrmap e4: x(active, since 2s) 2023-06-22T17:19:53.521 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2023-06-22T17:19:53.521 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout { 2023-06-22T17:19:53.522 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "fsid": "d88002e2-1120-11ee-9b2b-001a4aab830c", 2023-06-22T17:19:53.522 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "health": { 2023-06-22T17:19:53.522 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-06-22T17:19:53.522 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-06-22T17:19:53.522 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-06-22T17:19:53.522 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2023-06-22T17:19:53.523 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-06-22T17:19:53.523 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-06-22T17:19:53.523 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 0 2023-06-22T17:19:53.523 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2023-06-22T17:19:53.523 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-06-22T17:19:53.523 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "a" 2023-06-22T17:19:53.523 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2023-06-22T17:19:53.523 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2023-06-22T17:19:53.524 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "monmap": { 2023-06-22T17:19:53.524 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-06-22T17:19:53.524 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2023-06-22T17:19:53.524 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-06-22T17:19:53.524 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2023-06-22T17:19:53.524 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-06-22T17:19:53.524 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-06-22T17:19:53.525 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-06-22T17:19:53.525 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-06-22T17:19:53.525 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-06-22T17:19:53.525 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-06-22T17:19:53.525 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-06-22T17:19:53.525 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-06-22T17:19:53.525 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2023-06-22T17:19:53.526 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-06-22T17:19:53.526 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-06-22T17:19:53.526 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-06-22T17:19:53.526 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-06-22T17:19:53.526 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-06-22T17:19:53.527 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-06-22T17:19:53.527 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-06-22T17:19:53.527 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-06-22T17:19:53.527 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-06-22T17:19:53.527 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2023-06-22T17:19:53.527 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-06-22T17:19:53.527 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-06-22T17:19:53.528 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-06-22T17:19:53.528 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-06-22T17:19:53.528 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2023-06-22T17:19:53.528 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-06-22T17:19:53.528 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "available": true, 2023-06-22T17:19:53.528 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-06-22T17:19:53.528 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "modules": [ 2023-06-22T17:19:53.529 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "iostat", 2023-06-22T17:19:53.529 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "nfs", 2023-06-22T17:19:53.529 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "restful" 2023-06-22T17:19:53.529 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2023-06-22T17:19:53.529 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "services": {} 2023-06-22T17:19:53.529 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2023-06-22T17:19:53.529 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-06-22T17:19:53.529 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-06-22T17:19:53.530 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "modified": "2023-06-22T17:19:36.314220+0000", 2023-06-22T17:19:53.530 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "services": {} 2023-06-22T17:19:53.530 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2023-06-22T17:19:53.530 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-06-22T17:19:53.530 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout } 2023-06-22T17:19:53.530 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.682+0000 7fe1c868d700 1 Processor -- start 2023-06-22T17:19:53.530 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.682+0000 7fe1c868d700 1 -- start start 2023-06-22T17:19:53.530 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.682+0000 7fe1c868d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1c0101320 0x7fe1c0101730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:19:53.531 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.682+0000 7fe1c868d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c0101c70 con 0x7fe1c0101320 2023-06-22T17:19:53.531 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.682+0000 7fe1c6429700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1c0101320 0x7fe1c0101730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:19:53.531 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.682+0000 7fe1c6429700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1c0101320 0x7fe1c0101730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:60530/0 (socket says 172.21.15.39:60530) 2023-06-22T17:19:53.531 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.682+0000 7fe1c6429700 1 -- 172.21.15.39:0/57705981 learned_addr learned my addr 172.21.15.39:0/57705981 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:19:53.531 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.683+0000 7fe1c6429700 1 -- 172.21.15.39:0/57705981 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c0101db0 con 0x7fe1c0101320 2023-06-22T17:19:53.531 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.683+0000 7fe1c6429700 1 --2- 172.21.15.39:0/57705981 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1c0101320 0x7fe1c0101730 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc0098e0 tx=0x7fe1bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e498b0ac36aee07 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:19:53.531 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.683+0000 7fe1c5427700 1 -- 172.21.15.39:0/57705981 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe1bc009e80 con 0x7fe1c0101320 2023-06-22T17:19:53.531 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.683+0000 7fe1c5427700 1 -- 172.21.15.39:0/57705981 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe1bc00bd30 con 0x7fe1c0101320 2023-06-22T17:19:53.532 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.683+0000 7fe1c5427700 1 -- 172.21.15.39:0/57705981 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe1bc011da0 con 0x7fe1c0101320 2023-06-22T17:19:53.532 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.684+0000 7fe1c868d700 1 -- 172.21.15.39:0/57705981 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1c0101320 msgr2=0x7fe1c0101730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:19:53.532 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.684+0000 7fe1c868d700 1 --2- 172.21.15.39:0/57705981 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1c0101320 0x7fe1c0101730 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc0098e0 tx=0x7fe1bc009250 comp rx=0 tx=0).stop 2023-06-22T17:19:53.532 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.684+0000 7fe1c868d700 1 -- 172.21.15.39:0/57705981 shutdown_connections 2023-06-22T17:19:53.532 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.684+0000 7fe1c868d700 1 --2- 172.21.15.39:0/57705981 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1c0101320 0x7fe1c0101730 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:19:53.532 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.684+0000 7fe1c868d700 1 -- 172.21.15.39:0/57705981 >> 172.21.15.39:0/57705981 conn(0x7fe1c00fcac0 msgr2=0x7fe1c00feed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:19:53.532 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.684+0000 7fe1c868d700 1 -- 172.21.15.39:0/57705981 shutdown_connections 2023-06-22T17:19:53.533 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.684+0000 7fe1c868d700 1 -- 172.21.15.39:0/57705981 wait complete. 2023-06-22T17:19:53.533 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.684+0000 7fe1c868d700 1 Processor -- start 2023-06-22T17:19:53.535 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.684+0000 7fe1c868d700 1 -- start start 2023-06-22T17:19:53.536 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.684+0000 7fe1c868d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1c0101320 0x7fe1c00769b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:19:53.536 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.685+0000 7fe1c868d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c0113ee0 con 0x7fe1c0101320 2023-06-22T17:19:53.536 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.685+0000 7fe1c6429700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1c0101320 0x7fe1c00769b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:19:53.536 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.685+0000 7fe1c6429700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1c0101320 0x7fe1c00769b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:60542/0 (socket says 172.21.15.39:60542) 2023-06-22T17:19:53.537 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.685+0000 7fe1c6429700 1 -- 172.21.15.39:0/3983313361 learned_addr learned my addr 172.21.15.39:0/3983313361 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:19:53.537 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.685+0000 7fe1c6429700 1 -- 172.21.15.39:0/3983313361 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1bc0094f0 con 0x7fe1c0101320 2023-06-22T17:19:53.537 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.685+0000 7fe1c6429700 1 --2- 172.21.15.39:0/3983313361 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1c0101320 0x7fe1c00769b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc010640 tx=0x7fe1bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:19:53.537 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.685+0000 7fe1b6ffd700 1 -- 172.21.15.39:0/3983313361 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe1bc010b90 con 0x7fe1c0101320 2023-06-22T17:19:53.537 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.685+0000 7fe1b6ffd700 1 -- 172.21.15.39:0/3983313361 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe1bc010cf0 con 0x7fe1c0101320 2023-06-22T17:19:53.538 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.685+0000 7fe1c868d700 1 -- 172.21.15.39:0/3983313361 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1c0114020 con 0x7fe1c0101320 2023-06-22T17:19:53.538 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.685+0000 7fe1b6ffd700 1 -- 172.21.15.39:0/3983313361 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe1bc022400 con 0x7fe1c0101320 2023-06-22T17:19:53.538 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.685+0000 7fe1c868d700 1 -- 172.21.15.39:0/3983313361 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1c0114370 con 0x7fe1c0101320 2023-06-22T17:19:53.538 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.686+0000 7fe1b6ffd700 1 -- 172.21.15.39:0/3983313361 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 3) v1 ==== 43102+0+0 (secure 0 0 0) 0x7fe1bc01a430 con 0x7fe1c0101320 2023-06-22T17:19:53.538 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.686+0000 7fe1c868d700 1 -- 172.21.15.39:0/3983313361 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1c005f060 con 0x7fe1c0101320 2023-06-22T17:19:53.538 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.686+0000 7fe1b6ffd700 1 --2- 172.21.15.39:0/3983313361 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7fe1ac0364a0 0x7fe1ac038950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:19:53.539 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.686+0000 7fe1b6ffd700 1 -- 172.21.15.39:0/3983313361 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fe1bc04a630 con 0x7fe1c0101320 2023-06-22T17:19:53.539 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.686+0000 7fe1c5c28700 1 --2- 172.21.15.39:0/3983313361 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7fe1ac0364a0 0x7fe1ac038950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:19:53.539 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.687+0000 7fe1c5c28700 1 --2- 172.21.15.39:0/3983313361 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7fe1ac0364a0 0x7fe1ac038950 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0009990 tx=0x7fe1b0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:19:53.539 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.691+0000 7fe1b6ffd700 1 -- 172.21.15.39:0/3983313361 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fe1bc02ab80 con 0x7fe1c0101320 2023-06-22T17:19:53.539 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.963+0000 7fe1c868d700 1 -- 172.21.15.39:0/3983313361 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fe1c0114620 con 0x7fe1c0101320 2023-06-22T17:19:53.539 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.964+0000 7fe1b6ffd700 1 -- 172.21.15.39:0/3983313361 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (secure 0 0 0) 0x7fe1bc02ab80 con 0x7fe1c0101320 2023-06-22T17:19:53.539 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.966+0000 7fe1c868d700 1 -- 172.21.15.39:0/3983313361 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7fe1ac0364a0 msgr2=0x7fe1ac038950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:19:53.539 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.966+0000 7fe1c868d700 1 --2- 172.21.15.39:0/3983313361 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7fe1ac0364a0 0x7fe1ac038950 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0009990 tx=0x7fe1b0006e40 comp rx=0 tx=0).stop 2023-06-22T17:19:53.540 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.966+0000 7fe1c868d700 1 -- 172.21.15.39:0/3983313361 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1c0101320 msgr2=0x7fe1c00769b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:19:53.540 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.966+0000 7fe1c868d700 1 --2- 172.21.15.39:0/3983313361 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1c0101320 0x7fe1c00769b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc010640 tx=0x7fe1bc009250 comp rx=0 tx=0).stop 2023-06-22T17:19:53.540 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.966+0000 7fe1c868d700 1 -- 172.21.15.39:0/3983313361 shutdown_connections 2023-06-22T17:19:53.540 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.966+0000 7fe1c868d700 1 --2- 172.21.15.39:0/3983313361 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7fe1ac0364a0 0x7fe1ac038950 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:19:53.540 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.966+0000 7fe1c868d700 1 --2- 172.21.15.39:0/3983313361 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1c0101320 0x7fe1c00769b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:19:53.540 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.966+0000 7fe1c868d700 1 -- 172.21.15.39:0/3983313361 >> 172.21.15.39:0/3983313361 conn(0x7fe1c00fcac0 msgr2=0x7fe1c00fd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:19:53.541 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.966+0000 7fe1c868d700 1 -- 172.21.15.39:0/3983313361 shutdown_connections 2023-06-22T17:19:53.541 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:50.966+0000 7fe1c868d700 1 -- 172.21.15.39:0/3983313361 wait complete. 2023-06-22T17:19:53.541 INFO:teuthology.orchestra.run.smithi039.stdout:mgr is available 2023-06-22T17:19:56.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:56 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/3720546394' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-06-22T17:19:57.675 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2023-06-22T17:19:57.676 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout [global] 2023-06-22T17:19:57.676 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout fsid = d88002e2-1120-11ee-9b2b-001a4aab830c 2023-06-22T17:19:57.676 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-06-22T17:19:57.676 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-06-22T17:19:57.676 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-06-22T17:19:57.677 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-06-22T17:19:57.677 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2023-06-22T17:19:57.677 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout [mgr] 2023-06-22T17:19:57.677 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-06-22T17:19:57.677 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2023-06-22T17:19:57.677 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout [osd] 2023-06-22T17:19:57.677 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-06-22T17:19:57.677 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-06-22T17:19:57.678 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-06-22T17:19:57.678 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-06-22T17:19:57.678 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-06-22T17:19:57.678 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.580+0000 7f8bbfd42700 1 Processor -- start 2023-06-22T17:19:57.678 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.580+0000 7f8bbfd42700 1 -- start start 2023-06-22T17:19:57.678 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.580+0000 7f8bbfd42700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8bb8101300 0x7f8bb8101710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:19:57.678 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.580+0000 7f8bbfd42700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb8101c50 con 0x7f8bb8101300 2023-06-22T17:19:57.678 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.580+0000 7f8bbdade700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8bb8101300 0x7f8bb8101710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:19:57.679 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.580+0000 7f8bbdade700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8bb8101300 0x7f8bb8101710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:60550/0 (socket says 172.21.15.39:60550) 2023-06-22T17:19:57.679 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.580+0000 7f8bbdade700 1 -- 172.21.15.39:0/2899329111 learned_addr learned my addr 172.21.15.39:0/2899329111 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:19:57.679 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.581+0000 7f8bbdade700 1 -- 172.21.15.39:0/2899329111 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb8101d90 con 0x7f8bb8101300 2023-06-22T17:19:57.679 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.581+0000 7f8bbdade700 1 --2- 172.21.15.39:0/2899329111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8bb8101300 0x7f8bb8101710 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8bb400bb80 tx=0x7f8bb400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=733ede5ffbf92635 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:19:57.679 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.581+0000 7f8bbcadc700 1 -- 172.21.15.39:0/2899329111 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8bb400dba0 con 0x7f8bb8101300 2023-06-22T17:19:57.679 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.581+0000 7f8bbcadc700 1 -- 172.21.15.39:0/2899329111 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8bb400dd00 con 0x7f8bb8101300 2023-06-22T17:19:57.679 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.581+0000 7f8bbcadc700 1 -- 172.21.15.39:0/2899329111 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8bb40187f0 con 0x7f8bb8101300 2023-06-22T17:19:57.680 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.582+0000 7f8bbfd42700 1 -- 172.21.15.39:0/2899329111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8bb8101300 msgr2=0x7f8bb8101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:19:57.680 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.582+0000 7f8bbfd42700 1 --2- 172.21.15.39:0/2899329111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8bb8101300 0x7f8bb8101710 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8bb400bb80 tx=0x7f8bb400b4f0 comp rx=0 tx=0).stop 2023-06-22T17:19:57.680 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.582+0000 7f8bbfd42700 1 -- 172.21.15.39:0/2899329111 shutdown_connections 2023-06-22T17:19:57.680 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.582+0000 7f8bbfd42700 1 --2- 172.21.15.39:0/2899329111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8bb8101300 0x7f8bb8101710 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:19:57.680 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.582+0000 7f8bbfd42700 1 -- 172.21.15.39:0/2899329111 >> 172.21.15.39:0/2899329111 conn(0x7f8bb80fca60 msgr2=0x7f8bb80fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:19:57.680 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.582+0000 7f8bbfd42700 1 -- 172.21.15.39:0/2899329111 shutdown_connections 2023-06-22T17:19:57.680 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.582+0000 7f8bbfd42700 1 -- 172.21.15.39:0/2899329111 wait complete. 2023-06-22T17:19:57.681 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.582+0000 7f8bbfd42700 1 Processor -- start 2023-06-22T17:19:57.681 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.582+0000 7f8bbfd42700 1 -- start start 2023-06-22T17:19:57.681 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.582+0000 7f8bbfd42700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8bb8101300 0x7f8bb810ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:19:57.681 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.582+0000 7f8bbfd42700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb810f370 con 0x7f8bb8101300 2023-06-22T17:19:57.681 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.583+0000 7f8bbdade700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8bb8101300 0x7f8bb810ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:19:57.681 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.583+0000 7f8bbdade700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8bb8101300 0x7f8bb810ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:60566/0 (socket says 172.21.15.39:60566) 2023-06-22T17:19:57.681 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.583+0000 7f8bbdade700 1 -- 172.21.15.39:0/3720546394 learned_addr learned my addr 172.21.15.39:0/3720546394 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:19:57.682 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.583+0000 7f8bbdade700 1 -- 172.21.15.39:0/3720546394 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb400b790 con 0x7f8bb8101300 2023-06-22T17:19:57.682 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.583+0000 7f8bbdade700 1 --2- 172.21.15.39:0/3720546394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8bb8101300 0x7f8bb810ee30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4006b00 tx=0x7f8bb400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:19:57.682 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.583+0000 7f8bae7fc700 1 -- 172.21.15.39:0/3720546394 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8bb400dba0 con 0x7f8bb8101300 2023-06-22T17:19:57.682 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.583+0000 7f8bae7fc700 1 -- 172.21.15.39:0/3720546394 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8bb401d050 con 0x7f8bb8101300 2023-06-22T17:19:57.682 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.583+0000 7f8bae7fc700 1 -- 172.21.15.39:0/3720546394 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8bb4022570 con 0x7f8bb8101300 2023-06-22T17:19:57.682 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.583+0000 7f8bbfd42700 1 -- 172.21.15.39:0/3720546394 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bb810f570 con 0x7f8bb8101300 2023-06-22T17:19:57.682 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.583+0000 7f8bbfd42700 1 -- 172.21.15.39:0/3720546394 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bb810f980 con 0x7f8bb8101300 2023-06-22T17:19:57.683 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.584+0000 7f8bae7fc700 1 -- 172.21.15.39:0/3720546394 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 4) v1 ==== 43208+0+0 (secure 0 0 0) 0x7f8bb40227a0 con 0x7f8bb8101300 2023-06-22T17:19:57.683 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.584+0000 7f8bae7fc700 1 --2- 172.21.15.39:0/3720546394 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7f8ba4036560 0x7f8ba4038a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:19:57.683 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.584+0000 7f8bae7fc700 1 -- 172.21.15.39:0/3720546394 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f8bb4014070 con 0x7f8bb8101300 2023-06-22T17:19:57.683 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.584+0000 7f8bbfd42700 1 -- 172.21.15.39:0/3720546394 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bb81087e0 con 0x7f8bb8101300 2023-06-22T17:19:57.683 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.584+0000 7f8bbd2dd700 1 --2- 172.21.15.39:0/3720546394 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7f8ba4036560 0x7f8ba4038a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:19:57.683 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.585+0000 7f8bbd2dd700 1 --2- 172.21.15.39:0/3720546394 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7f8ba4036560 0x7f8ba4038a10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8009990 tx=0x7f8ba8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:19:57.684 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.589+0000 7f8bae7fc700 1 -- 172.21.15.39:0/3720546394 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f8bb4022a50 con 0x7f8bb8101300 2023-06-22T17:19:57.684 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.785+0000 7f8bbfd42700 1 -- 172.21.15.39:0/3720546394 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f8bb805f060 con 0x7f8bb8101300 2023-06-22T17:19:57.684 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.786+0000 7f8bae7fc700 1 -- 172.21.15.39:0/3720546394 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f8bb4021b20 con 0x7f8bb8101300 2023-06-22T17:19:57.684 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.788+0000 7f8bbfd42700 1 -- 172.21.15.39:0/3720546394 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7f8ba4036560 msgr2=0x7f8ba4038a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:19:57.684 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.788+0000 7f8bbfd42700 1 --2- 172.21.15.39:0/3720546394 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7f8ba4036560 0x7f8ba4038a10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8009990 tx=0x7f8ba8006e40 comp rx=0 tx=0).stop 2023-06-22T17:19:57.684 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.788+0000 7f8bbfd42700 1 -- 172.21.15.39:0/3720546394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8bb8101300 msgr2=0x7f8bb810ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:19:57.684 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.788+0000 7f8bbfd42700 1 --2- 172.21.15.39:0/3720546394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8bb8101300 0x7f8bb810ee30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4006b00 tx=0x7f8bb400b4f0 comp rx=0 tx=0).stop 2023-06-22T17:19:57.685 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.788+0000 7f8bbfd42700 1 -- 172.21.15.39:0/3720546394 shutdown_connections 2023-06-22T17:19:57.685 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.788+0000 7f8bbfd42700 1 --2- 172.21.15.39:0/3720546394 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7f8ba4036560 0x7f8ba4038a10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:19:57.685 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.788+0000 7f8bbfd42700 1 --2- 172.21.15.39:0/3720546394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8bb8101300 0x7f8bb810ee30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:19:57.685 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.788+0000 7f8bbfd42700 1 -- 172.21.15.39:0/3720546394 >> 172.21.15.39:0/3720546394 conn(0x7f8bb80fca60 msgr2=0x7f8bb80fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:19:57.685 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.788+0000 7f8bbfd42700 1 -- 172.21.15.39:0/3720546394 shutdown_connections 2023-06-22T17:19:57.685 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:55.788+0000 7f8bbfd42700 1 -- 172.21.15.39:0/3720546394 wait complete. 2023-06-22T17:19:57.685 INFO:teuthology.orchestra.run.smithi039.stdout:Enabling cephadm module... 2023-06-22T17:19:58.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:58 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/2003493048' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-06-22T17:19:58.682 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:19:58 smithi039 conmon[85217]: ignoring --setuser ceph since I am not root 2023-06-22T17:19:58.683 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:19:58 smithi039 conmon[85217]: ignoring --setgroup ceph since I am not root 2023-06-22T17:19:58.820 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.111+0000 7f8ea15ae700 1 Processor -- start 2023-06-22T17:19:58.821 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.111+0000 7f8ea15ae700 1 -- start start 2023-06-22T17:19:58.821 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.111+0000 7f8ea15ae700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e9c101320 0x7f8e9c101730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:19:58.821 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.111+0000 7f8ea15ae700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9c101c70 con 0x7f8e9c101320 2023-06-22T17:19:58.821 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.111+0000 7f8e9affd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e9c101320 0x7f8e9c101730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:19:58.821 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.111+0000 7f8e9affd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e9c101320 0x7f8e9c101730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39778/0 (socket says 172.21.15.39:39778) 2023-06-22T17:19:58.822 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.111+0000 7f8e9affd700 1 -- 172.21.15.39:0/3124535505 learned_addr learned my addr 172.21.15.39:0/3124535505 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:19:58.822 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.112+0000 7f8e9affd700 1 -- 172.21.15.39:0/3124535505 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e9c101db0 con 0x7f8e9c101320 2023-06-22T17:19:58.822 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.112+0000 7f8e9affd700 1 --2- 172.21.15.39:0/3124535505 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e9c101320 0x7f8e9c101730 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c00bb80 tx=0x7f8e8c00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5030055bc0d9642 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:19:58.822 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.112+0000 7f8e99ffb700 1 -- 172.21.15.39:0/3124535505 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e8c00dba0 con 0x7f8e9c101320 2023-06-22T17:19:58.822 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.112+0000 7f8e99ffb700 1 -- 172.21.15.39:0/3124535505 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8e8c00dd00 con 0x7f8e9c101320 2023-06-22T17:19:58.822 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.112+0000 7f8e99ffb700 1 -- 172.21.15.39:0/3124535505 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e8c0187f0 con 0x7f8e9c101320 2023-06-22T17:19:58.823 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.112+0000 7f8ea15ae700 1 -- 172.21.15.39:0/3124535505 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e9c101320 msgr2=0x7f8e9c101730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:19:58.823 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.112+0000 7f8ea15ae700 1 --2- 172.21.15.39:0/3124535505 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e9c101320 0x7f8e9c101730 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c00bb80 tx=0x7f8e8c00b4f0 comp rx=0 tx=0).stop 2023-06-22T17:19:58.823 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.112+0000 7f8ea15ae700 1 -- 172.21.15.39:0/3124535505 shutdown_connections 2023-06-22T17:19:58.823 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.112+0000 7f8ea15ae700 1 --2- 172.21.15.39:0/3124535505 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e9c101320 0x7f8e9c101730 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:19:58.823 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.112+0000 7f8ea15ae700 1 -- 172.21.15.39:0/3124535505 >> 172.21.15.39:0/3124535505 conn(0x7f8e9c0fcac0 msgr2=0x7f8e9c0feed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:19:58.823 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.112+0000 7f8ea15ae700 1 -- 172.21.15.39:0/3124535505 shutdown_connections 2023-06-22T17:19:58.823 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.113+0000 7f8ea15ae700 1 -- 172.21.15.39:0/3124535505 wait complete. 2023-06-22T17:19:58.823 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.113+0000 7f8ea15ae700 1 Processor -- start 2023-06-22T17:19:58.824 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.113+0000 7f8ea15ae700 1 -- start start 2023-06-22T17:19:58.824 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.113+0000 7f8ea15ae700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e9c101320 0x7f8e9c10ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:19:58.824 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.113+0000 7f8ea15ae700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9c10f3b0 con 0x7f8e9c101320 2023-06-22T17:19:58.824 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.114+0000 7f8e9affd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e9c101320 0x7f8e9c10ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:19:58.824 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.114+0000 7f8e9affd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e9c101320 0x7f8e9c10ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39782/0 (socket says 172.21.15.39:39782) 2023-06-22T17:19:58.824 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.114+0000 7f8e9affd700 1 -- 172.21.15.39:0/2003493048 learned_addr learned my addr 172.21.15.39:0/2003493048 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:19:58.824 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.114+0000 7f8e9affd700 1 -- 172.21.15.39:0/2003493048 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e8c00b790 con 0x7f8e9c101320 2023-06-22T17:19:58.825 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.114+0000 7f8e9affd700 1 --2- 172.21.15.39:0/2003493048 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e9c101320 0x7f8e9c10ee70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c00dc70 tx=0x7f8e8c00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:19:58.825 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.114+0000 7f8e837fe700 1 -- 172.21.15.39:0/2003493048 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e8c0108a0 con 0x7f8e9c101320 2023-06-22T17:19:58.825 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.114+0000 7f8e837fe700 1 -- 172.21.15.39:0/2003493048 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8e8c01d050 con 0x7f8e9c101320 2023-06-22T17:19:58.825 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.114+0000 7f8ea15ae700 1 -- 172.21.15.39:0/2003493048 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e9c10f5b0 con 0x7f8e9c101320 2023-06-22T17:19:58.825 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.114+0000 7f8e837fe700 1 -- 172.21.15.39:0/2003493048 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e8c022570 con 0x7f8e9c101320 2023-06-22T17:19:58.825 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.114+0000 7f8ea15ae700 1 -- 172.21.15.39:0/2003493048 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e9c10f9c0 con 0x7f8e9c101320 2023-06-22T17:19:58.825 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.115+0000 7f8e837fe700 1 -- 172.21.15.39:0/2003493048 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 4) v1 ==== 43208+0+0 (secure 0 0 0) 0x7f8e8c023470 con 0x7f8e9c101320 2023-06-22T17:19:58.826 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.115+0000 7f8e837fe700 1 --2- 172.21.15.39:0/2003493048 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7f8e840365c0 0x7f8e84038a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:19:58.826 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.115+0000 7f8ea15ae700 1 -- 172.21.15.39:0/2003493048 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e9c108800 con 0x7f8e9c101320 2023-06-22T17:19:58.826 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.115+0000 7f8e837fe700 1 -- 172.21.15.39:0/2003493048 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f8e8c014070 con 0x7f8e9c101320 2023-06-22T17:19:58.826 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.115+0000 7f8e9a7fc700 1 --2- 172.21.15.39:0/2003493048 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7f8e840365c0 0x7f8e84038a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:19:58.826 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.116+0000 7f8e9a7fc700 1 --2- 172.21.15.39:0/2003493048 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7f8e840365c0 0x7f8e84038a70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8e90009990 tx=0x7f8e90006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:19:58.826 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.120+0000 7f8e837fe700 1 -- 172.21.15.39:0/2003493048 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f8e8c022720 con 0x7f8e9c101320 2023-06-22T17:19:58.827 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.367+0000 7f8ea15ae700 1 -- 172.21.15.39:0/2003493048 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f8e9c05f060 con 0x7f8e9c101320 2023-06-22T17:19:58.827 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.420+0000 7f8e837fe700 1 -- 172.21.15.39:0/2003493048 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mgrmap(e 5) v1 ==== 43219+0+0 (secure 0 0 0) 0x7f8e8c022aa0 con 0x7f8e9c101320 2023-06-22T17:19:58.827 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.421+0000 7f8e837fe700 1 -- 172.21.15.39:0/2003493048 <== mon.0 v2:172.21.15.39:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f8e8c0202d0 con 0x7f8e9c101320 2023-06-22T17:19:58.827 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.423+0000 7f8ea15ae700 1 -- 172.21.15.39:0/2003493048 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7f8e840365c0 msgr2=0x7f8e84038a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:19:58.827 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.423+0000 7f8ea15ae700 1 --2- 172.21.15.39:0/2003493048 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7f8e840365c0 0x7f8e84038a70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8e90009990 tx=0x7f8e90006e40 comp rx=0 tx=0).stop 2023-06-22T17:19:58.827 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.423+0000 7f8ea15ae700 1 -- 172.21.15.39:0/2003493048 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e9c101320 msgr2=0x7f8e9c10ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:19:58.827 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.423+0000 7f8ea15ae700 1 --2- 172.21.15.39:0/2003493048 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e9c101320 0x7f8e9c10ee70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c00dc70 tx=0x7f8e8c00b4f0 comp rx=0 tx=0).stop 2023-06-22T17:19:58.827 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.423+0000 7f8ea15ae700 1 -- 172.21.15.39:0/2003493048 shutdown_connections 2023-06-22T17:19:58.828 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.423+0000 7f8ea15ae700 1 --2- 172.21.15.39:0/2003493048 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7f8e840365c0 0x7f8e84038a70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:19:58.828 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.423+0000 7f8ea15ae700 1 --2- 172.21.15.39:0/2003493048 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e9c101320 0x7f8e9c10ee70 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:19:58.828 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.423+0000 7f8ea15ae700 1 -- 172.21.15.39:0/2003493048 >> 172.21.15.39:0/2003493048 conn(0x7f8e9c0fcac0 msgr2=0x7f8e9c0fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:19:58.828 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.423+0000 7f8ea15ae700 1 -- 172.21.15.39:0/2003493048 shutdown_connections 2023-06-22T17:19:58.828 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:58.424+0000 7f8ea15ae700 1 -- 172.21.15.39:0/2003493048 wait complete. 2023-06-22T17:19:59.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:59 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/2003493048' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-06-22T17:19:59.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:19:59 smithi039 ceph-mon[84805]: mgrmap e5: x(active, since 9s) 2023-06-22T17:19:59.960 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout { 2023-06-22T17:19:59.960 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-06-22T17:19:59.960 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "available": true, 2023-06-22T17:19:59.960 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-06-22T17:19:59.960 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-06-22T17:19:59.961 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout } 2023-06-22T17:19:59.961 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.299+0000 7f8e8d32a700 1 Processor -- start 2023-06-22T17:19:59.961 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.299+0000 7f8e8d32a700 1 -- start start 2023-06-22T17:19:59.961 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.299+0000 7f8e8d32a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e88070e00 0x7f8e88104fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:19:59.961 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.299+0000 7f8e8d32a700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e88105510 con 0x7f8e88070e00 2023-06-22T17:19:59.961 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.299+0000 7f8e86ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e88070e00 0x7f8e88104fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:19:59.961 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.299+0000 7f8e86ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e88070e00 0x7f8e88104fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39820/0 (socket says 172.21.15.39:39820) 2023-06-22T17:19:59.962 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.299+0000 7f8e86ffd700 1 -- 172.21.15.39:0/401843388 learned_addr learned my addr 172.21.15.39:0/401843388 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:19:59.962 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.300+0000 7f8e86ffd700 1 -- 172.21.15.39:0/401843388 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e88105650 con 0x7f8e88070e00 2023-06-22T17:19:59.962 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.300+0000 7f8e86ffd700 1 --2- 172.21.15.39:0/401843388 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e88070e00 0x7f8e88104fd0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8e780098e0 tx=0x7f8e78009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38bb23442924fccb server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:19:59.962 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.300+0000 7f8e85ffb700 1 -- 172.21.15.39:0/401843388 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e78009e80 con 0x7f8e88070e00 2023-06-22T17:19:59.962 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.300+0000 7f8e85ffb700 1 -- 172.21.15.39:0/401843388 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8e7800bd30 con 0x7f8e88070e00 2023-06-22T17:19:59.962 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.300+0000 7f8e85ffb700 1 -- 172.21.15.39:0/401843388 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e78011da0 con 0x7f8e88070e00 2023-06-22T17:19:59.962 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.300+0000 7f8e8d32a700 1 -- 172.21.15.39:0/401843388 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e88070e00 msgr2=0x7f8e88104fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:19:59.962 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.300+0000 7f8e8d32a700 1 --2- 172.21.15.39:0/401843388 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e88070e00 0x7f8e88104fd0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8e780098e0 tx=0x7f8e78009250 comp rx=0 tx=0).stop 2023-06-22T17:19:59.963 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.301+0000 7f8e8d32a700 1 -- 172.21.15.39:0/401843388 shutdown_connections 2023-06-22T17:19:59.963 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.301+0000 7f8e8d32a700 1 --2- 172.21.15.39:0/401843388 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e88070e00 0x7f8e88104fd0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:19:59.963 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.301+0000 7f8e8d32a700 1 -- 172.21.15.39:0/401843388 >> 172.21.15.39:0/401843388 conn(0x7f8e880fca60 msgr2=0x7f8e880fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:19:59.963 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.301+0000 7f8e8d32a700 1 -- 172.21.15.39:0/401843388 shutdown_connections 2023-06-22T17:19:59.963 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.301+0000 7f8e8d32a700 1 -- 172.21.15.39:0/401843388 wait complete. 2023-06-22T17:19:59.963 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.301+0000 7f8e8d32a700 1 Processor -- start 2023-06-22T17:19:59.963 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.301+0000 7f8e8d32a700 1 -- start start 2023-06-22T17:19:59.963 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.302+0000 7f8e8d32a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e88070e00 0x7f8e881110a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:19:59.964 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.302+0000 7f8e8d32a700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e881115e0 con 0x7f8e88070e00 2023-06-22T17:19:59.964 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.302+0000 7f8e86ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e88070e00 0x7f8e881110a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:19:59.964 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.302+0000 7f8e86ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e88070e00 0x7f8e881110a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39824/0 (socket says 172.21.15.39:39824) 2023-06-22T17:19:59.964 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.302+0000 7f8e86ffd700 1 -- 172.21.15.39:0/323138533 learned_addr learned my addr 172.21.15.39:0/323138533 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:19:59.964 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.302+0000 7f8e86ffd700 1 -- 172.21.15.39:0/323138533 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e780094f0 con 0x7f8e88070e00 2023-06-22T17:19:59.964 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.302+0000 7f8e86ffd700 1 --2- 172.21.15.39:0/323138533 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e88070e00 0x7f8e881110a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8e78006fd0 tx=0x7f8e78009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:19:59.964 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.302+0000 7f8e6f7fe700 1 -- 172.21.15.39:0/323138533 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e78010b30 con 0x7f8e88070e00 2023-06-22T17:19:59.965 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.302+0000 7f8e8d32a700 1 -- 172.21.15.39:0/323138533 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e881117e0 con 0x7f8e88070e00 2023-06-22T17:19:59.965 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.302+0000 7f8e6f7fe700 1 -- 172.21.15.39:0/323138533 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8e78010c90 con 0x7f8e88070e00 2023-06-22T17:19:59.965 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.302+0000 7f8e6f7fe700 1 -- 172.21.15.39:0/323138533 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e78022400 con 0x7f8e88070e00 2023-06-22T17:19:59.965 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.302+0000 7f8e8d32a700 1 -- 172.21.15.39:0/323138533 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e88111bf0 con 0x7f8e88070e00 2023-06-22T17:19:59.965 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.303+0000 7f8e8d32a700 1 -- 172.21.15.39:0/323138533 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e88106500 con 0x7f8e88070e00 2023-06-22T17:19:59.965 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.304+0000 7f8e6f7fe700 1 -- 172.21.15.39:0/323138533 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 43219+0+0 (secure 0 0 0) 0x7f8e7801a430 con 0x7f8e88070e00 2023-06-22T17:19:59.965 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.304+0000 7f8e6f7fe700 1 --2- 172.21.15.39:0/323138533 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7f8e70036560 0x7f8e70038a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:19:59.965 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.304+0000 7f8e6f7fe700 1 -- 172.21.15.39:0/323138533 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f8e7804ad10 con 0x7f8e88070e00 2023-06-22T17:19:59.966 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.304+0000 7f8e867fc700 1 -- 172.21.15.39:0/323138533 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7f8e70036560 msgr2=0x7f8e70038a10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/3960151496 2023-06-22T17:19:59.966 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.304+0000 7f8e867fc700 1 --2- 172.21.15.39:0/323138533 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7f8e70036560 0x7f8e70038a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T17:19:59.966 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.312+0000 7f8e6f7fe700 1 -- 172.21.15.39:0/323138533 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f8e78021d70 con 0x7f8e88070e00 2023-06-22T17:19:59.966 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.505+0000 7f8e867fc700 1 -- 172.21.15.39:0/323138533 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7f8e70036560 msgr2=0x7f8e70038a10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/3960151496 2023-06-22T17:19:59.966 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.505+0000 7f8e867fc700 1 --2- 172.21.15.39:0/323138533 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7f8e70036560 0x7f8e70038a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T17:19:59.966 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.563+0000 7f8e8d32a700 1 -- 172.21.15.39:0/323138533 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f8e88114740 con 0x7f8e88070e00 2023-06-22T17:19:59.966 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.563+0000 7f8e6f7fe700 1 -- 172.21.15.39:0/323138533 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f8e88114740 con 0x7f8e88070e00 2023-06-22T17:19:59.967 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.565+0000 7f8e8d32a700 1 -- 172.21.15.39:0/323138533 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7f8e70036560 msgr2=0x7f8e70038a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-06-22T17:19:59.967 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.565+0000 7f8e8d32a700 1 --2- 172.21.15.39:0/323138533 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7f8e70036560 0x7f8e70038a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:19:59.967 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.566+0000 7f8e8d32a700 1 -- 172.21.15.39:0/323138533 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e88070e00 msgr2=0x7f8e881110a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:19:59.967 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.566+0000 7f8e8d32a700 1 --2- 172.21.15.39:0/323138533 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e88070e00 0x7f8e881110a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8e78006fd0 tx=0x7f8e78009250 comp rx=0 tx=0).stop 2023-06-22T17:19:59.967 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.566+0000 7f8e8d32a700 1 -- 172.21.15.39:0/323138533 shutdown_connections 2023-06-22T17:19:59.967 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.566+0000 7f8e8d32a700 1 --2- 172.21.15.39:0/323138533 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7f8e70036560 0x7f8e70038a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:19:59.967 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.566+0000 7f8e8d32a700 1 --2- 172.21.15.39:0/323138533 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e88070e00 0x7f8e881110a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:19:59.967 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.566+0000 7f8e8d32a700 1 -- 172.21.15.39:0/323138533 >> 172.21.15.39:0/323138533 conn(0x7f8e880fca60 msgr2=0x7f8e880fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:19:59.968 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.566+0000 7f8e8d32a700 1 -- 172.21.15.39:0/323138533 shutdown_connections 2023-06-22T17:19:59.968 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:19:59.566+0000 7f8e8d32a700 1 -- 172.21.15.39:0/323138533 wait complete. 2023-06-22T17:19:59.968 INFO:teuthology.orchestra.run.smithi039.stdout:Waiting for the mgr to restart... 2023-06-22T17:19:59.968 INFO:teuthology.orchestra.run.smithi039.stdout:Waiting for mgr epoch 5... 2023-06-22T17:20:00.010 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:19:59 smithi039 conmon[85217]: 2023-06-22T17:19:59.741+0000 7f65d43651c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-06-22T17:20:00.011 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:19:59 smithi039 conmon[85217]: 2023-06-22T17:19:59.910+0000 7f65d43651c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-06-22T17:20:00.431 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:00 smithi039 conmon[85217]: 2023-06-22T17:20:00.009+0000 7f65d43651c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-06-22T17:20:00.432 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:00 smithi039 conmon[85217]: 2023-06-22T17:20:00.244+0000 7f65d43651c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-06-22T17:20:00.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:00 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/323138533' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-06-22T17:20:00.682 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:00 smithi039 conmon[85217]: 2023-06-22T17:20:00.466+0000 7f65d43651c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-06-22T17:20:00.682 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:00 smithi039 conmon[85217]: 2023-06-22T17:20:00.631+0000 7f65d43651c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-06-22T17:20:01.182 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:00 smithi039 conmon[85217]: 2023-06-22T17:20:00.714+0000 7f65d43651c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-06-22T17:20:01.682 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:01 smithi039 conmon[85217]: 2023-06-22T17:20:01.324+0000 7f65d43651c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-06-22T17:20:02.182 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:01 smithi039 conmon[85217]: 2023-06-22T17:20:01.830+0000 7f65d43651c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-06-22T17:20:02.182 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:01 smithi039 conmon[85217]: 2023-06-22T17:20:01.926+0000 7f65d43651c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-06-22T17:20:02.182 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:02 smithi039 conmon[85217]: 2023-06-22T17:20:02.022+0000 7f65d43651c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-06-22T17:20:02.916 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:02 smithi039 conmon[85217]: 2023-06-22T17:20:02.641+0000 7f65d43651c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-06-22T17:20:02.916 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:02 smithi039 conmon[85217]: 2023-06-22T17:20:02.819+0000 7f65d43651c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-06-22T17:20:03.182 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:02 smithi039 conmon[85217]: 2023-06-22T17:20:02.915+0000 7f65d43651c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-06-22T17:20:03.182 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:03 smithi039 conmon[85217]: 2023-06-22T17:20:03.050+0000 7f65d43651c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-06-22T17:20:03.182 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:03 smithi039 conmon[85217]: 2023-06-22T17:20:03.144+0000 7f65d43651c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-06-22T17:20:04.403 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:04 smithi039 conmon[85217]: 2023-06-22T17:20:04.063+0000 7f65d43651c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-06-22T17:20:04.403 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:04 smithi039 conmon[85217]: 2023-06-22T17:20:04.156+0000 7f65d43651c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-06-22T17:20:04.404 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:04 smithi039 conmon[85217]: 2023-06-22T17:20:04.247+0000 7f65d43651c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-06-22T17:20:04.682 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:04 smithi039 conmon[85217]: 2023-06-22T17:20:04.402+0000 7f65d43651c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-06-22T17:20:04.682 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:04 smithi039 conmon[85217]: 2023-06-22T17:20:04.505+0000 7f65d43651c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-06-22T17:20:05.182 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:04 smithi039 conmon[85217]: 2023-06-22T17:20:04.914+0000 7f65d43651c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-06-22T17:20:05.182 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:05 smithi039 conmon[85217]: 2023-06-22T17:20:05.008+0000 7f65d43651c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-06-22T17:20:05.182 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:05 smithi039 conmon[85217]: 2023-06-22T17:20:05.101+0000 7f65d43651c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-06-22T17:20:05.682 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:05 smithi039 conmon[85217]: 2023-06-22T17:20:05.193+0000 7f65d43651c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-06-22T17:20:05.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:05 smithi039 ceph-mon[84805]: Active manager daemon x restarted 2023-06-22T17:20:05.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:05 smithi039 ceph-mon[84805]: Activating manager daemon x 2023-06-22T17:20:05.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:05 smithi039 ceph-mon[84805]: osdmap e2: 0 total, 0 up, 0 in 2023-06-22T17:20:05.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:05 smithi039 ceph-mon[84805]: mgrmap e6: x(active, starting, since 0.00314314s) 2023-06-22T17:20:05.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:05 smithi039 ceph-mon[84805]: from='mgr.14118 172.21.15.39:0/1180832241' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-22T17:20:05.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:05 smithi039 ceph-mon[84805]: from='mgr.14118 172.21.15.39:0/1180832241' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-06-22T17:20:05.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:05 smithi039 ceph-mon[84805]: from='mgr.14118 172.21.15.39:0/1180832241' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-06-22T17:20:05.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:05 smithi039 ceph-mon[84805]: from='mgr.14118 172.21.15.39:0/1180832241' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-06-22T17:20:05.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:05 smithi039 ceph-mon[84805]: from='mgr.14118 172.21.15.39:0/1180832241' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-06-22T17:20:05.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:05 smithi039 ceph-mon[84805]: Manager daemon x is now available 2023-06-22T17:20:05.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:05 smithi039 ceph-mon[84805]: from='mgr.14118 172.21.15.39:0/1180832241' entity='mgr.x' 2023-06-22T17:20:05.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:05 smithi039 ceph-mon[84805]: from='mgr.14118 172.21.15.39:0/1180832241' entity='mgr.x' 2023-06-22T17:20:05.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:05 smithi039 ceph-mon[84805]: from='mgr.14118 172.21.15.39:0/1180832241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:20:05.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:05 smithi039 ceph-mon[84805]: from='mgr.14118 172.21.15.39:0/1180832241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:20:06.573 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:06 smithi039 ceph-mon[84805]: Found migration_current of "None". Setting to last migration. 2023-06-22T17:20:06.574 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:06 smithi039 ceph-mon[84805]: from='mgr.14118 172.21.15.39:0/1180832241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:20:06.574 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:06 smithi039 ceph-mon[84805]: from='mgr.14118 172.21.15.39:0/1180832241' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-06-22T17:20:06.574 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:06 smithi039 ceph-mon[84805]: from='mgr.14118 172.21.15.39:0/1180832241' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-06-22T17:20:06.574 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:06 smithi039 ceph-mon[84805]: mgrmap e7: x(active, since 1.00501s) 2023-06-22T17:20:06.603 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout { 2023-06-22T17:20:06.608 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-06-22T17:20:06.608 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "initialized": true 2023-06-22T17:20:06.608 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout } 2023-06-22T17:20:06.609 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.451+0000 7efca298b700 1 Processor -- start 2023-06-22T17:20:06.609 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.451+0000 7efca298b700 1 -- start start 2023-06-22T17:20:06.609 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.451+0000 7efca298b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc9c070e00 0x7efc9c104fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:06.609 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.452+0000 7efca298b700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c105510 con 0x7efc9c070e00 2023-06-22T17:20:06.609 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.452+0000 7efc9bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc9c070e00 0x7efc9c104fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:06.609 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.452+0000 7efc9bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc9c070e00 0x7efc9c104fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39832/0 (socket says 172.21.15.39:39832) 2023-06-22T17:20:06.609 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.452+0000 7efc9bfff700 1 -- 172.21.15.39:0/1906974876 learned_addr learned my addr 172.21.15.39:0/1906974876 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:06.610 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.452+0000 7efc9bfff700 1 -- 172.21.15.39:0/1906974876 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc9c105650 con 0x7efc9c070e00 2023-06-22T17:20:06.610 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.452+0000 7efc9bfff700 1 --2- 172.21.15.39:0/1906974876 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc9c070e00 0x7efc9c104fd0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efc8c0098e0 tx=0x7efc8c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9354f6dc10e45243 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:06.610 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.452+0000 7efc9affd700 1 -- 172.21.15.39:0/1906974876 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efc8c009e80 con 0x7efc9c070e00 2023-06-22T17:20:06.610 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.452+0000 7efc9affd700 1 -- 172.21.15.39:0/1906974876 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efc8c00bd30 con 0x7efc9c070e00 2023-06-22T17:20:06.610 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.452+0000 7efc9affd700 1 -- 172.21.15.39:0/1906974876 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efc8c011da0 con 0x7efc9c070e00 2023-06-22T17:20:06.610 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.453+0000 7efca298b700 1 -- 172.21.15.39:0/1906974876 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc9c070e00 msgr2=0x7efc9c104fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:06.610 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.453+0000 7efca298b700 1 --2- 172.21.15.39:0/1906974876 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc9c070e00 0x7efc9c104fd0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efc8c0098e0 tx=0x7efc8c009250 comp rx=0 tx=0).stop 2023-06-22T17:20:06.610 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.453+0000 7efca298b700 1 -- 172.21.15.39:0/1906974876 shutdown_connections 2023-06-22T17:20:06.611 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.453+0000 7efca298b700 1 --2- 172.21.15.39:0/1906974876 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc9c070e00 0x7efc9c104fd0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:06.611 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.453+0000 7efca298b700 1 -- 172.21.15.39:0/1906974876 >> 172.21.15.39:0/1906974876 conn(0x7efc9c0fca60 msgr2=0x7efc9c0fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:06.611 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.453+0000 7efca298b700 1 -- 172.21.15.39:0/1906974876 shutdown_connections 2023-06-22T17:20:06.611 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.453+0000 7efca298b700 1 -- 172.21.15.39:0/1906974876 wait complete. 2023-06-22T17:20:06.611 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.454+0000 7efca298b700 1 Processor -- start 2023-06-22T17:20:06.611 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.454+0000 7efca298b700 1 -- start start 2023-06-22T17:20:06.611 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.454+0000 7efca298b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc9c070e00 0x7efc9c10ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:06.611 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.454+0000 7efca298b700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c10f3c0 con 0x7efc9c070e00 2023-06-22T17:20:06.612 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.454+0000 7efc9bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc9c070e00 0x7efc9c10ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:06.612 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.454+0000 7efc9bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc9c070e00 0x7efc9c10ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39834/0 (socket says 172.21.15.39:39834) 2023-06-22T17:20:06.612 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.454+0000 7efc9bfff700 1 -- 172.21.15.39:0/566363817 learned_addr learned my addr 172.21.15.39:0/566363817 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:06.612 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.454+0000 7efc9bfff700 1 -- 172.21.15.39:0/566363817 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc8c0094f0 con 0x7efc9c070e00 2023-06-22T17:20:06.612 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.455+0000 7efc9bfff700 1 --2- 172.21.15.39:0/566363817 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc9c070e00 0x7efc9c10ee80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efc8c006fd0 tx=0x7efc8c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:06.613 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.455+0000 7efc98ff9700 1 -- 172.21.15.39:0/566363817 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efc8c010b30 con 0x7efc9c070e00 2023-06-22T17:20:06.613 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.455+0000 7efc98ff9700 1 -- 172.21.15.39:0/566363817 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efc8c010c90 con 0x7efc9c070e00 2023-06-22T17:20:06.613 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.455+0000 7efca298b700 1 -- 172.21.15.39:0/566363817 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc9c10f5c0 con 0x7efc9c070e00 2023-06-22T17:20:06.613 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.455+0000 7efc98ff9700 1 -- 172.21.15.39:0/566363817 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efc8c022400 con 0x7efc9c070e00 2023-06-22T17:20:06.613 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.455+0000 7efca298b700 1 -- 172.21.15.39:0/566363817 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc9c10f9d0 con 0x7efc9c070e00 2023-06-22T17:20:06.613 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.456+0000 7efc98ff9700 1 -- 172.21.15.39:0/566363817 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 43219+0+0 (secure 0 0 0) 0x7efc8c022670 con 0x7efc9c070e00 2023-06-22T17:20:06.614 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.456+0000 7efc98ff9700 1 --2- 172.21.15.39:0/566363817 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7efc8803aa00 0x7efc8803ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:06.614 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.456+0000 7efc98ff9700 1 -- 172.21.15.39:0/566363817 --> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc8803d5c0 con 0x7efc8803aa00 2023-06-22T17:20:06.614 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.456+0000 7efc98ff9700 1 -- 172.21.15.39:0/566363817 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7efc8c04a980 con 0x7efc9c070e00 2023-06-22T17:20:06.614 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.456+0000 7efc9b7fe700 1 -- 172.21.15.39:0/566363817 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7efc8803aa00 msgr2=0x7efc8803ceb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/3960151496 2023-06-22T17:20:06.614 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.456+0000 7efc9b7fe700 1 --2- 172.21.15.39:0/566363817 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7efc8803aa00 0x7efc8803ceb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T17:20:06.614 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.657+0000 7efc9b7fe700 1 -- 172.21.15.39:0/566363817 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7efc8803aa00 msgr2=0x7efc8803ceb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/3960151496 2023-06-22T17:20:06.614 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:00.657+0000 7efc9b7fe700 1 --2- 172.21.15.39:0/566363817 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7efc8803aa00 0x7efc8803ceb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T17:20:06.615 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:01.058+0000 7efc9b7fe700 1 -- 172.21.15.39:0/566363817 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7efc8803aa00 msgr2=0x7efc8803ceb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/3960151496 2023-06-22T17:20:06.615 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:01.058+0000 7efc9b7fe700 1 --2- 172.21.15.39:0/566363817 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7efc8803aa00 0x7efc8803ceb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T17:20:06.615 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:01.859+0000 7efc9b7fe700 1 -- 172.21.15.39:0/566363817 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7efc8803aa00 msgr2=0x7efc8803ceb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/3960151496 2023-06-22T17:20:06.615 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:01.859+0000 7efc9b7fe700 1 --2- 172.21.15.39:0/566363817 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7efc8803aa00 0x7efc8803ceb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T17:20:06.615 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:03.460+0000 7efc9b7fe700 1 -- 172.21.15.39:0/566363817 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7efc8803aa00 msgr2=0x7efc8803ceb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/3960151496 2023-06-22T17:20:06.615 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:03.460+0000 7efc9b7fe700 1 --2- 172.21.15.39:0/566363817 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7efc8803aa00 0x7efc8803ceb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-06-22T17:20:06.615 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:05.208+0000 7efc98ff9700 1 -- 172.21.15.39:0/566363817 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mgrmap(e 6) v1 ==== 42986+0+0 (secure 0 0 0) 0x7efc8c022ba0 con 0x7efc9c070e00 2023-06-22T17:20:06.615 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:05.208+0000 7efc98ff9700 1 -- 172.21.15.39:0/566363817 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7efc8803aa00 msgr2=0x7efc8803ceb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-06-22T17:20:06.616 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:05.208+0000 7efc98ff9700 1 --2- 172.21.15.39:0/566363817 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7efc8803aa00 0x7efc8803ceb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:06.616 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:06.209+0000 7efc98ff9700 1 -- 172.21.15.39:0/566363817 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mgrmap(e 7) v1 ==== 43113+0+0 (secure 0 0 0) 0x7efc8c04bdd0 con 0x7efc9c070e00 2023-06-22T17:20:06.616 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:06.209+0000 7efc98ff9700 1 --2- 172.21.15.39:0/566363817 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7efc8803e1f0 0x7efc880405d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:06.616 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:06.209+0000 7efc98ff9700 1 -- 172.21.15.39:0/566363817 --> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc8803d5c0 con 0x7efc8803e1f0 2023-06-22T17:20:06.616 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:06.209+0000 7efc9b7fe700 1 --2- 172.21.15.39:0/566363817 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7efc8803e1f0 0x7efc880405d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:06.616 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:06.210+0000 7efc9b7fe700 1 --2- 172.21.15.39:0/566363817 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7efc8803e1f0 0x7efc880405d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7efc840039d0 tx=0x7efc84009350 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:06.616 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:06.210+0000 7efc98ff9700 1 -- 172.21.15.39:0/566363817 <== mgr.14118 v2:172.21.15.39:6800/1350590065 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7efc8803d5c0 con 0x7efc8803e1f0 2023-06-22T17:20:06.617 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:06.216+0000 7efca298b700 1 -- 172.21.15.39:0/566363817 --> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7efc9c1013b0 con 0x7efc8803e1f0 2023-06-22T17:20:06.617 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:06.216+0000 7efc98ff9700 1 -- 172.21.15.39:0/566363817 <== mgr.14118 v2:172.21.15.39:6800/1350590065 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7efc9c1013b0 con 0x7efc8803e1f0 2023-06-22T17:20:06.617 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:06.217+0000 7efca298b700 1 -- 172.21.15.39:0/566363817 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7efc8803e1f0 msgr2=0x7efc880405d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:06.617 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:06.217+0000 7efca298b700 1 --2- 172.21.15.39:0/566363817 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7efc8803e1f0 0x7efc880405d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7efc840039d0 tx=0x7efc84009350 comp rx=0 tx=0).stop 2023-06-22T17:20:06.617 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:06.217+0000 7efca298b700 1 -- 172.21.15.39:0/566363817 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc9c070e00 msgr2=0x7efc9c10ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:06.617 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:06.217+0000 7efca298b700 1 --2- 172.21.15.39:0/566363817 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc9c070e00 0x7efc9c10ee80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efc8c006fd0 tx=0x7efc8c009250 comp rx=0 tx=0).stop 2023-06-22T17:20:06.617 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:06.217+0000 7efca298b700 1 -- 172.21.15.39:0/566363817 shutdown_connections 2023-06-22T17:20:06.617 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:06.217+0000 7efca298b700 1 --2- 172.21.15.39:0/566363817 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7efc8803e1f0 0x7efc880405d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:06.618 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:06.217+0000 7efca298b700 1 --2- 172.21.15.39:0/566363817 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc9c070e00 0x7efc9c10ee80 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:06.618 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:06.217+0000 7efca298b700 1 --2- 172.21.15.39:0/566363817 >> [v2:172.21.15.39:6800/3960151496,v1:172.21.15.39:6801/3960151496] conn(0x7efc8803aa00 0x7efc8803ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:06.618 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:06.217+0000 7efca298b700 1 -- 172.21.15.39:0/566363817 >> 172.21.15.39:0/566363817 conn(0x7efc9c0fca60 msgr2=0x7efc9c102c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:06.618 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:06.217+0000 7efca298b700 1 -- 172.21.15.39:0/566363817 shutdown_connections 2023-06-22T17:20:06.618 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:06.217+0000 7efca298b700 1 -- 172.21.15.39:0/566363817 wait complete. 2023-06-22T17:20:06.618 INFO:teuthology.orchestra.run.smithi039.stdout:mgr epoch 5 is available 2023-06-22T17:20:06.618 INFO:teuthology.orchestra.run.smithi039.stdout:Setting orchestrator backend to cephadm... 2023-06-22T17:20:07.648 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.066+0000 7f2ca9ce1700 1 Processor -- start 2023-06-22T17:20:07.648 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.066+0000 7f2ca9ce1700 1 -- start start 2023-06-22T17:20:07.648 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.066+0000 7f2ca9ce1700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ca4101300 0x7f2ca4101710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:07.649 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.066+0000 7f2ca9ce1700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca4101c50 con 0x7f2ca4101300 2023-06-22T17:20:07.649 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.066+0000 7f2ca37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ca4101300 0x7f2ca4101710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:07.649 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.066+0000 7f2ca37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ca4101300 0x7f2ca4101710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33512/0 (socket says 172.21.15.39:33512) 2023-06-22T17:20:07.649 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.066+0000 7f2ca37fe700 1 -- 172.21.15.39:0/2755078442 learned_addr learned my addr 172.21.15.39:0/2755078442 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:07.649 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.066+0000 7f2ca37fe700 1 -- 172.21.15.39:0/2755078442 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca4101d90 con 0x7f2ca4101300 2023-06-22T17:20:07.649 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.067+0000 7f2ca37fe700 1 --2- 172.21.15.39:0/2755078442 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ca4101300 0x7f2ca4101710 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2c940098e0 tx=0x7f2c94009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d5f173c2662135d server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:07.649 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.067+0000 7f2ca27fc700 1 -- 172.21.15.39:0/2755078442 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2c94009e80 con 0x7f2ca4101300 2023-06-22T17:20:07.649 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.067+0000 7f2ca27fc700 1 -- 172.21.15.39:0/2755078442 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2c9400bd30 con 0x7f2ca4101300 2023-06-22T17:20:07.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.067+0000 7f2ca27fc700 1 -- 172.21.15.39:0/2755078442 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2c94011da0 con 0x7f2ca4101300 2023-06-22T17:20:07.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.068+0000 7f2ca9ce1700 1 -- 172.21.15.39:0/2755078442 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ca4101300 msgr2=0x7f2ca4101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:07.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.068+0000 7f2ca9ce1700 1 --2- 172.21.15.39:0/2755078442 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ca4101300 0x7f2ca4101710 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2c940098e0 tx=0x7f2c94009250 comp rx=0 tx=0).stop 2023-06-22T17:20:07.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.068+0000 7f2ca9ce1700 1 -- 172.21.15.39:0/2755078442 shutdown_connections 2023-06-22T17:20:07.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.068+0000 7f2ca9ce1700 1 --2- 172.21.15.39:0/2755078442 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ca4101300 0x7f2ca4101710 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:07.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.068+0000 7f2ca9ce1700 1 -- 172.21.15.39:0/2755078442 >> 172.21.15.39:0/2755078442 conn(0x7f2ca40fca60 msgr2=0x7f2ca40fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:07.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.068+0000 7f2ca9ce1700 1 -- 172.21.15.39:0/2755078442 shutdown_connections 2023-06-22T17:20:07.651 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.068+0000 7f2ca9ce1700 1 -- 172.21.15.39:0/2755078442 wait complete. 2023-06-22T17:20:07.651 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.068+0000 7f2ca9ce1700 1 Processor -- start 2023-06-22T17:20:07.651 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.068+0000 7f2ca9ce1700 1 -- start start 2023-06-22T17:20:07.651 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.068+0000 7f2ca9ce1700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ca4101300 0x7f2ca410ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:07.651 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.068+0000 7f2ca9ce1700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca41104d0 con 0x7f2ca4101300 2023-06-22T17:20:07.651 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.069+0000 7f2ca37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ca4101300 0x7f2ca410ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:07.651 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.069+0000 7f2ca37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ca4101300 0x7f2ca410ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33522/0 (socket says 172.21.15.39:33522) 2023-06-22T17:20:07.651 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.069+0000 7f2ca37fe700 1 -- 172.21.15.39:0/1611289216 learned_addr learned my addr 172.21.15.39:0/1611289216 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:07.652 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.069+0000 7f2ca37fe700 1 -- 172.21.15.39:0/1611289216 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c940094f0 con 0x7f2ca4101300 2023-06-22T17:20:07.652 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.069+0000 7f2ca37fe700 1 --2- 172.21.15.39:0/1611289216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ca4101300 0x7f2ca410ff90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2c94009fd0 tx=0x7f2c94009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:07.652 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.069+0000 7f2c8bfff700 1 -- 172.21.15.39:0/1611289216 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2c94010b60 con 0x7f2ca4101300 2023-06-22T17:20:07.652 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.069+0000 7f2ca9ce1700 1 -- 172.21.15.39:0/1611289216 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ca41106d0 con 0x7f2ca4101300 2023-06-22T17:20:07.652 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.069+0000 7f2c8bfff700 1 -- 172.21.15.39:0/1611289216 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2c94010cc0 con 0x7f2ca4101300 2023-06-22T17:20:07.652 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.069+0000 7f2c8bfff700 1 -- 172.21.15.39:0/1611289216 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2c94022400 con 0x7f2ca4101300 2023-06-22T17:20:07.652 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.069+0000 7f2ca9ce1700 1 -- 172.21.15.39:0/1611289216 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ca4110ae0 con 0x7f2ca4101300 2023-06-22T17:20:07.653 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.070+0000 7f2c8bfff700 1 -- 172.21.15.39:0/1611289216 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 7) v1 ==== 43113+0+0 (secure 0 0 0) 0x7f2c940226b0 con 0x7f2ca4101300 2023-06-22T17:20:07.653 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.070+0000 7f2ca9ce1700 1 -- 172.21.15.39:0/1611289216 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ca41031b0 con 0x7f2ca4101300 2023-06-22T17:20:07.653 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.070+0000 7f2c8bfff700 1 --2- 172.21.15.39:0/1611289216 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f2c8c0364a0 0x7f2c8c038950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:07.653 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.070+0000 7f2c8bfff700 1 -- 172.21.15.39:0/1611289216 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f2c94010e30 con 0x7f2ca4101300 2023-06-22T17:20:07.653 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.072+0000 7f2ca2ffd700 1 --2- 172.21.15.39:0/1611289216 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f2c8c0364a0 0x7f2c8c038950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:07.653 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.072+0000 7f2ca2ffd700 1 --2- 172.21.15.39:0/1611289216 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f2c8c0364a0 0x7f2c8c038950 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2c98009990 tx=0x7f2c98006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:07.653 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.075+0000 7f2c8bfff700 1 -- 172.21.15.39:0/1611289216 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f2c94018350 con 0x7f2ca4101300 2023-06-22T17:20:07.653 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.273+0000 7f2c8bfff700 1 -- 172.21.15.39:0/1611289216 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mgrmap(e 8) v1 ==== 43219+0+0 (secure 0 0 0) 0x7f2c9402c960 con 0x7f2ca4101300 2023-06-22T17:20:07.654 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.286+0000 7f2ca9ce1700 1 -- 172.21.15.39:0/1611289216 --> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ca40fdf20 con 0x7f2c8c0364a0 2023-06-22T17:20:07.654 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.292+0000 7f2c8bfff700 1 -- 172.21.15.39:0/1611289216 <== mgr.14118 v2:172.21.15.39:6800/1350590065 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2ca40fdf20 con 0x7f2c8c0364a0 2023-06-22T17:20:07.654 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.294+0000 7f2ca9ce1700 1 -- 172.21.15.39:0/1611289216 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f2c8c0364a0 msgr2=0x7f2c8c038950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:07.654 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.294+0000 7f2ca9ce1700 1 --2- 172.21.15.39:0/1611289216 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f2c8c0364a0 0x7f2c8c038950 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2c98009990 tx=0x7f2c98006e40 comp rx=0 tx=0).stop 2023-06-22T17:20:07.654 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.294+0000 7f2ca9ce1700 1 -- 172.21.15.39:0/1611289216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ca4101300 msgr2=0x7f2ca410ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:07.654 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.294+0000 7f2ca9ce1700 1 --2- 172.21.15.39:0/1611289216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ca4101300 0x7f2ca410ff90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2c94009fd0 tx=0x7f2c94009250 comp rx=0 tx=0).stop 2023-06-22T17:20:07.654 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.294+0000 7f2ca9ce1700 1 -- 172.21.15.39:0/1611289216 shutdown_connections 2023-06-22T17:20:07.654 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.294+0000 7f2ca9ce1700 1 --2- 172.21.15.39:0/1611289216 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f2c8c0364a0 0x7f2c8c038950 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:07.655 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.294+0000 7f2ca9ce1700 1 --2- 172.21.15.39:0/1611289216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ca4101300 0x7f2ca410ff90 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:07.655 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.294+0000 7f2ca9ce1700 1 -- 172.21.15.39:0/1611289216 >> 172.21.15.39:0/1611289216 conn(0x7f2ca40fca60 msgr2=0x7f2ca40fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:07.655 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.295+0000 7f2ca9ce1700 1 -- 172.21.15.39:0/1611289216 shutdown_connections 2023-06-22T17:20:07.655 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:07.295+0000 7f2ca9ce1700 1 -- 172.21.15.39:0/1611289216 wait complete. 2023-06-22T17:20:07.655 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:07 smithi039 ceph-mon[84805]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-06-22T17:20:07.656 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:07 smithi039 ceph-mon[84805]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-06-22T17:20:08.526 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:08 smithi039 ceph-mon[84805]: mgrmap e8: x(active, since 2s) 2023-06-22T17:20:08.527 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:08 smithi039 ceph-mon[84805]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:20:08.527 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:08 smithi039 ceph-mon[84805]: from='mgr.14118 172.21.15.39:0/1180832241' entity='mgr.x' 2023-06-22T17:20:08.527 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:08 smithi039 ceph-mon[84805]: from='mgr.14118 172.21.15.39:0/1180832241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:20:08.691 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout value unchanged 2023-06-22T17:20:08.691 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.114+0000 7fda9c015700 1 Processor -- start 2023-06-22T17:20:08.692 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.114+0000 7fda9c015700 1 -- start start 2023-06-22T17:20:08.692 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.115+0000 7fda9c015700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda94070dc0 0x7fda94105010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:08.692 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.115+0000 7fda9c015700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda94105550 con 0x7fda94070dc0 2023-06-22T17:20:08.693 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.115+0000 7fda99db1700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda94070dc0 0x7fda94105010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:08.693 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.115+0000 7fda99db1700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda94070dc0 0x7fda94105010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33524/0 (socket says 172.21.15.39:33524) 2023-06-22T17:20:08.693 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.115+0000 7fda99db1700 1 -- 172.21.15.39:0/2605595162 learned_addr learned my addr 172.21.15.39:0/2605595162 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:08.693 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.115+0000 7fda99db1700 1 -- 172.21.15.39:0/2605595162 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda94105690 con 0x7fda94070dc0 2023-06-22T17:20:08.694 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.115+0000 7fda99db1700 1 --2- 172.21.15.39:0/2605595162 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda94070dc0 0x7fda94105010 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fda840098e0 tx=0x7fda84009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ec499f1ff9dc45a server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:08.694 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.115+0000 7fda98daf700 1 -- 172.21.15.39:0/2605595162 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fda84009e80 con 0x7fda94070dc0 2023-06-22T17:20:08.694 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.115+0000 7fda98daf700 1 -- 172.21.15.39:0/2605595162 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fda8400bd30 con 0x7fda94070dc0 2023-06-22T17:20:08.695 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.115+0000 7fda98daf700 1 -- 172.21.15.39:0/2605595162 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fda84011da0 con 0x7fda94070dc0 2023-06-22T17:20:08.695 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.116+0000 7fda9c015700 1 -- 172.21.15.39:0/2605595162 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda94070dc0 msgr2=0x7fda94105010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:08.695 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.116+0000 7fda9c015700 1 --2- 172.21.15.39:0/2605595162 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda94070dc0 0x7fda94105010 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fda840098e0 tx=0x7fda84009250 comp rx=0 tx=0).stop 2023-06-22T17:20:08.695 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.116+0000 7fda9c015700 1 -- 172.21.15.39:0/2605595162 shutdown_connections 2023-06-22T17:20:08.696 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.116+0000 7fda9c015700 1 --2- 172.21.15.39:0/2605595162 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda94070dc0 0x7fda94105010 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:08.696 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.116+0000 7fda9c015700 1 -- 172.21.15.39:0/2605595162 >> 172.21.15.39:0/2605595162 conn(0x7fda940fca60 msgr2=0x7fda940fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:08.696 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.116+0000 7fda9c015700 1 -- 172.21.15.39:0/2605595162 shutdown_connections 2023-06-22T17:20:08.696 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.116+0000 7fda9c015700 1 -- 172.21.15.39:0/2605595162 wait complete. 2023-06-22T17:20:08.696 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.117+0000 7fda9c015700 1 Processor -- start 2023-06-22T17:20:08.697 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.117+0000 7fda9c015700 1 -- start start 2023-06-22T17:20:08.697 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.117+0000 7fda9c015700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda94070dc0 0x7fda941110c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:08.697 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.117+0000 7fda9c015700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda94111600 con 0x7fda94070dc0 2023-06-22T17:20:08.697 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.117+0000 7fda99db1700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda94070dc0 0x7fda941110c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:08.698 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.117+0000 7fda99db1700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda94070dc0 0x7fda941110c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33532/0 (socket says 172.21.15.39:33532) 2023-06-22T17:20:08.698 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.117+0000 7fda99db1700 1 -- 172.21.15.39:0/2967004552 learned_addr learned my addr 172.21.15.39:0/2967004552 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:08.698 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.117+0000 7fda99db1700 1 -- 172.21.15.39:0/2967004552 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda840094f0 con 0x7fda94070dc0 2023-06-22T17:20:08.698 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.117+0000 7fda99db1700 1 --2- 172.21.15.39:0/2967004552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda94070dc0 0x7fda941110c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fda84006fd0 tx=0x7fda84009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:08.699 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.118+0000 7fda8a7fc700 1 -- 172.21.15.39:0/2967004552 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fda84010b30 con 0x7fda94070dc0 2023-06-22T17:20:08.699 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.118+0000 7fda8a7fc700 1 -- 172.21.15.39:0/2967004552 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fda84010c90 con 0x7fda94070dc0 2023-06-22T17:20:08.699 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.118+0000 7fda9c015700 1 -- 172.21.15.39:0/2967004552 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda94111800 con 0x7fda94070dc0 2023-06-22T17:20:08.699 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.118+0000 7fda8a7fc700 1 -- 172.21.15.39:0/2967004552 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fda84022400 con 0x7fda94070dc0 2023-06-22T17:20:08.699 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.118+0000 7fda9c015700 1 -- 172.21.15.39:0/2967004552 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda94111c10 con 0x7fda94070dc0 2023-06-22T17:20:08.699 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.118+0000 7fda8a7fc700 1 -- 172.21.15.39:0/2967004552 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 8) v1 ==== 43219+0+0 (secure 0 0 0) 0x7fda840226c0 con 0x7fda94070dc0 2023-06-22T17:20:08.699 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.119+0000 7fda8a7fc700 1 --2- 172.21.15.39:0/2967004552 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7fda80036560 0x7fda80038a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:08.700 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.119+0000 7fda8a7fc700 1 -- 172.21.15.39:0/2967004552 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fda8404ad00 con 0x7fda94070dc0 2023-06-22T17:20:08.700 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.119+0000 7fda9c015700 1 -- 172.21.15.39:0/2967004552 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda9405f060 con 0x7fda94070dc0 2023-06-22T17:20:08.700 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.119+0000 7fda995b0700 1 --2- 172.21.15.39:0/2967004552 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7fda80036560 0x7fda80038a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:08.700 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.119+0000 7fda995b0700 1 --2- 172.21.15.39:0/2967004552 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7fda80036560 0x7fda80038a10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fda90009990 tx=0x7fda90006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:08.700 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.128+0000 7fda8a7fc700 1 -- 172.21.15.39:0/2967004552 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fda84018350 con 0x7fda94070dc0 2023-06-22T17:20:08.700 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.347+0000 7fda9c015700 1 -- 172.21.15.39:0/2967004552 --> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fda940fdf60 con 0x7fda80036560 2023-06-22T17:20:08.701 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.347+0000 7fda8a7fc700 1 -- 172.21.15.39:0/2967004552 <== mgr.14118 v2:172.21.15.39:6800/1350590065 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fda940fdf60 con 0x7fda80036560 2023-06-22T17:20:08.701 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.349+0000 7fda9c015700 1 -- 172.21.15.39:0/2967004552 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7fda80036560 msgr2=0x7fda80038a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:08.701 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.349+0000 7fda9c015700 1 --2- 172.21.15.39:0/2967004552 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7fda80036560 0x7fda80038a10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fda90009990 tx=0x7fda90006e40 comp rx=0 tx=0).stop 2023-06-22T17:20:08.701 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.349+0000 7fda9c015700 1 -- 172.21.15.39:0/2967004552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda94070dc0 msgr2=0x7fda941110c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:08.701 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.349+0000 7fda9c015700 1 --2- 172.21.15.39:0/2967004552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda94070dc0 0x7fda941110c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fda84006fd0 tx=0x7fda84009250 comp rx=0 tx=0).stop 2023-06-22T17:20:08.701 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.349+0000 7fda9c015700 1 -- 172.21.15.39:0/2967004552 shutdown_connections 2023-06-22T17:20:08.701 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.349+0000 7fda9c015700 1 --2- 172.21.15.39:0/2967004552 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7fda80036560 0x7fda80038a10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:08.701 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.349+0000 7fda9c015700 1 --2- 172.21.15.39:0/2967004552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda94070dc0 0x7fda941110c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:08.702 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.349+0000 7fda9c015700 1 -- 172.21.15.39:0/2967004552 >> 172.21.15.39:0/2967004552 conn(0x7fda940fca60 msgr2=0x7fda940fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:08.702 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.350+0000 7fda9c015700 1 -- 172.21.15.39:0/2967004552 shutdown_connections 2023-06-22T17:20:08.702 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:08.350+0000 7fda9c015700 1 -- 172.21.15.39:0/2967004552 wait complete. 2023-06-22T17:20:08.702 INFO:teuthology.orchestra.run.smithi039.stdout:Generating ssh key... 2023-06-22T17:20:09.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:09 smithi039 ceph-mon[84805]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:20:09.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:09 smithi039 ceph-mon[84805]: [22/Jun/2023:17:20:08] ENGINE Bus STARTING 2023-06-22T17:20:09.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:09 smithi039 ceph-mon[84805]: from='mgr.14118 172.21.15.39:0/1180832241' entity='mgr.x' 2023-06-22T17:20:09.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:09 smithi039 ceph-mon[84805]: from='mgr.14118 172.21.15.39:0/1180832241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:20:10.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:10 smithi039 ceph-mon[84805]: [22/Jun/2023:17:20:08] ENGINE Serving on https://172.21.15.39:7150 2023-06-22T17:20:10.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:10 smithi039 ceph-mon[84805]: [22/Jun/2023:17:20:08] ENGINE Bus STARTED 2023-06-22T17:20:10.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:10 smithi039 ceph-mon[84805]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:20:10.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:10 smithi039 ceph-mon[84805]: Generating ssh key... 2023-06-22T17:20:10.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:10 smithi039 ceph-mon[84805]: from='mgr.14118 172.21.15.39:0/1180832241' entity='mgr.x' 2023-06-22T17:20:10.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:10 smithi039 ceph-mon[84805]: from='mgr.14118 172.21.15.39:0/1180832241' entity='mgr.x' 2023-06-22T17:20:10.434 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:10 smithi039 conmon[85217]: Generating public/private rsa key pair. 2023-06-22T17:20:10.434 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:10 smithi039 conmon[85217]: Your identification has been saved in /tmp/tmpn6xrfck7/key. 2023-06-22T17:20:10.434 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:10 smithi039 conmon[85217]: Your public key has been saved in /tmp/tmpn6xrfck7/key.pub. 2023-06-22T17:20:10.434 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:10 smithi039 conmon[85217]: The key fingerprint is: 2023-06-22T17:20:10.434 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:10 smithi039 conmon[85217]: SHA256:vzPLIv9323+WQRvD5V9EgiAfDYb4tVUlIX8hBdiAN/c ceph-d88002e2-1120-11ee-9b2b-001a4aab830c 2023-06-22T17:20:10.434 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:10 smithi039 conmon[85217]: The key's randomart image is: 2023-06-22T17:20:10.434 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:10 smithi039 conmon[85217]: +---[RSA 3072]----+ 2023-06-22T17:20:10.435 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:10 smithi039 conmon[85217]: | ...*===B*o| 2023-06-22T17:20:10.435 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:10 smithi039 conmon[85217]: | . .=.=+=.o+| 2023-06-22T17:20:10.435 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:10 smithi039 conmon[85217]: | . .ooo +oo| 2023-06-22T17:20:10.435 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:10 smithi039 conmon[85217]: | . . Eo| 2023-06-22T17:20:10.435 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:10 smithi039 conmon[85217]: | S . *| 2023-06-22T17:20:10.435 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:10 smithi039 conmon[85217]: | . o.| 2023-06-22T17:20:10.435 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:10 smithi039 conmon[85217]: | . o| 2023-06-22T17:20:10.435 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:10 smithi039 conmon[85217]: | . ..o.. .oo| 2023-06-22T17:20:10.436 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:10 smithi039 conmon[85217]: | o.o== ..o=| 2023-06-22T17:20:10.436 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:10 smithi039 conmon[85217]: +----[SHA256]-----+ 2023-06-22T17:20:10.527 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.112+0000 7f9ba485d700 1 Processor -- start 2023-06-22T17:20:10.528 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.112+0000 7f9ba485d700 1 -- start start 2023-06-22T17:20:10.528 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.113+0000 7f9ba485d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ba0102900 0x7f9ba0102d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:10.528 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.113+0000 7f9ba485d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba0103250 con 0x7f9ba0102900 2023-06-22T17:20:10.528 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.113+0000 7f9b9e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ba0102900 0x7f9ba0102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:10.528 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.113+0000 7f9b9e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ba0102900 0x7f9ba0102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33534/0 (socket says 172.21.15.39:33534) 2023-06-22T17:20:10.528 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.113+0000 7f9b9e59c700 1 -- 172.21.15.39:0/1760127099 learned_addr learned my addr 172.21.15.39:0/1760127099 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:10.528 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.113+0000 7f9b9e59c700 1 -- 172.21.15.39:0/1760127099 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba0103390 con 0x7f9ba0102900 2023-06-22T17:20:10.529 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.113+0000 7f9b9e59c700 1 --2- 172.21.15.39:0/1760127099 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ba0102900 0x7f9ba0102d10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9b9000bb80 tx=0x7f9b9000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc500fe1a748afce server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:10.529 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.113+0000 7f9b9d59a700 1 -- 172.21.15.39:0/1760127099 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9b9000dba0 con 0x7f9ba0102900 2023-06-22T17:20:10.529 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.113+0000 7f9b9d59a700 1 -- 172.21.15.39:0/1760127099 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9b9000dd00 con 0x7f9ba0102900 2023-06-22T17:20:10.529 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.113+0000 7f9b9d59a700 1 -- 172.21.15.39:0/1760127099 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9b900187f0 con 0x7f9ba0102900 2023-06-22T17:20:10.529 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.114+0000 7f9ba485d700 1 -- 172.21.15.39:0/1760127099 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ba0102900 msgr2=0x7f9ba0102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:10.529 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.114+0000 7f9ba485d700 1 --2- 172.21.15.39:0/1760127099 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ba0102900 0x7f9ba0102d10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9b9000bb80 tx=0x7f9b9000b4f0 comp rx=0 tx=0).stop 2023-06-22T17:20:10.529 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.114+0000 7f9ba485d700 1 -- 172.21.15.39:0/1760127099 shutdown_connections 2023-06-22T17:20:10.529 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.114+0000 7f9ba485d700 1 --2- 172.21.15.39:0/1760127099 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ba0102900 0x7f9ba0102d10 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:10.530 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.114+0000 7f9ba485d700 1 -- 172.21.15.39:0/1760127099 >> 172.21.15.39:0/1760127099 conn(0x7f9ba00fc1e0 msgr2=0x7f9ba00fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:10.530 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.114+0000 7f9ba485d700 1 -- 172.21.15.39:0/1760127099 shutdown_connections 2023-06-22T17:20:10.530 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.114+0000 7f9ba485d700 1 -- 172.21.15.39:0/1760127099 wait complete. 2023-06-22T17:20:10.530 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.115+0000 7f9ba485d700 1 Processor -- start 2023-06-22T17:20:10.530 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.115+0000 7f9ba485d700 1 -- start start 2023-06-22T17:20:10.530 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.115+0000 7f9ba485d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ba0102900 0x7f9ba010ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:10.531 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.115+0000 7f9ba485d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba010f350 con 0x7f9ba0102900 2023-06-22T17:20:10.531 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.115+0000 7f9b9e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ba0102900 0x7f9ba010ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:10.531 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.115+0000 7f9b9e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ba0102900 0x7f9ba010ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33550/0 (socket says 172.21.15.39:33550) 2023-06-22T17:20:10.531 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.115+0000 7f9b9e59c700 1 -- 172.21.15.39:0/4222119363 learned_addr learned my addr 172.21.15.39:0/4222119363 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:10.531 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.115+0000 7f9b9e59c700 1 -- 172.21.15.39:0/4222119363 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b9000b790 con 0x7f9ba0102900 2023-06-22T17:20:10.531 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.115+0000 7f9b9e59c700 1 --2- 172.21.15.39:0/4222119363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ba0102900 0x7f9ba010ee10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9b90003080 tx=0x7f9b9000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:10.531 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.116+0000 7f9b8effd700 1 -- 172.21.15.39:0/4222119363 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9b90027070 con 0x7f9ba0102900 2023-06-22T17:20:10.531 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.116+0000 7f9b8effd700 1 -- 172.21.15.39:0/4222119363 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9b9001d050 con 0x7f9ba0102900 2023-06-22T17:20:10.532 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.116+0000 7f9ba485d700 1 -- 172.21.15.39:0/4222119363 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ba010f550 con 0x7f9ba0102900 2023-06-22T17:20:10.532 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.116+0000 7f9b8effd700 1 -- 172.21.15.39:0/4222119363 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9b900223d0 con 0x7f9ba0102900 2023-06-22T17:20:10.532 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.116+0000 7f9ba485d700 1 -- 172.21.15.39:0/4222119363 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ba010f960 con 0x7f9ba0102900 2023-06-22T17:20:10.532 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.116+0000 7f9b8effd700 1 -- 172.21.15.39:0/4222119363 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 8) v1 ==== 43219+0+0 (secure 0 0 0) 0x7f9b90023470 con 0x7f9ba0102900 2023-06-22T17:20:10.532 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.117+0000 7f9ba485d700 1 -- 172.21.15.39:0/4222119363 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ba005f060 con 0x7f9ba0102900 2023-06-22T17:20:10.532 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.117+0000 7f9b8effd700 1 --2- 172.21.15.39:0/4222119363 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f9b880365c0 0x7f9b88038a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:10.533 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.117+0000 7f9b8effd700 1 -- 172.21.15.39:0/4222119363 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f9b90014070 con 0x7f9ba0102900 2023-06-22T17:20:10.533 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.118+0000 7f9b9dd9b700 1 --2- 172.21.15.39:0/4222119363 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f9b880365c0 0x7f9b88038a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:10.533 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.118+0000 7f9b9dd9b700 1 --2- 172.21.15.39:0/4222119363 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f9b880365c0 0x7f9b88038a70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9b94009990 tx=0x7f9b94006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:10.533 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.121+0000 7f9b8effd700 1 -- 172.21.15.39:0/4222119363 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f9b9000e970 con 0x7f9ba0102900 2023-06-22T17:20:10.533 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:09.320+0000 7f9ba485d700 1 -- 172.21.15.39:0/4222119363 --> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ba002b1b0 con 0x7f9b880365c0 2023-06-22T17:20:10.533 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.152+0000 7f9b8effd700 1 -- 172.21.15.39:0/4222119363 <== mgr.14118 v2:172.21.15.39:6800/1350590065 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9ba002b1b0 con 0x7f9b880365c0 2023-06-22T17:20:10.533 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.154+0000 7f9ba485d700 1 -- 172.21.15.39:0/4222119363 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f9b880365c0 msgr2=0x7f9b88038a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:10.534 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.154+0000 7f9ba485d700 1 --2- 172.21.15.39:0/4222119363 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f9b880365c0 0x7f9b88038a70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9b94009990 tx=0x7f9b94006e40 comp rx=0 tx=0).stop 2023-06-22T17:20:10.534 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.154+0000 7f9ba485d700 1 -- 172.21.15.39:0/4222119363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ba0102900 msgr2=0x7f9ba010ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:10.534 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.154+0000 7f9ba485d700 1 --2- 172.21.15.39:0/4222119363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ba0102900 0x7f9ba010ee10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9b90003080 tx=0x7f9b9000b4f0 comp rx=0 tx=0).stop 2023-06-22T17:20:10.534 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.154+0000 7f9ba485d700 1 -- 172.21.15.39:0/4222119363 shutdown_connections 2023-06-22T17:20:10.534 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.154+0000 7f9ba485d700 1 --2- 172.21.15.39:0/4222119363 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f9b880365c0 0x7f9b88038a70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:10.534 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.154+0000 7f9ba485d700 1 --2- 172.21.15.39:0/4222119363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ba0102900 0x7f9ba010ee10 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:10.534 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.154+0000 7f9ba485d700 1 -- 172.21.15.39:0/4222119363 >> 172.21.15.39:0/4222119363 conn(0x7f9ba00fc1e0 msgr2=0x7f9ba01058a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:10.535 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.155+0000 7f9ba485d700 1 -- 172.21.15.39:0/4222119363 shutdown_connections 2023-06-22T17:20:10.535 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.155+0000 7f9ba485d700 1 -- 172.21.15.39:0/4222119363 wait complete. 2023-06-22T17:20:11.566 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5tQ9nDpBOSN+wtaWGNclkt7PkTanv9ZMInGcJeS0T8zZaeDxvKTlyvYR1jrtDQQxaF24a+cbnEn1sXhp7nu5lZ8smN/uwt2MCOO+yn0B2tCvMwBxmGjHnSxd5AyXscuIUituiObHRDirchwuGlZ3+HWqvlcAI2lXOrA/8GxXi0s9qqtkx7RBG8mNpWEZrUhibb9rl9vrQp9D4QGBzyiumUSJzGsd3VXXI8MavE6woeGMw1fH+seIDI8qYbqClG6y3+BOoLlZFspZqAE2TXT+fFXSzw571QCOQ5nWHNxZAiv5DwqqeYrbnIY++Veh4NLhl6vpVfgqqKlUIiEue7JVl+8zA9KHJQqLskT9moip5RO+3elKBpDa0hjcd7cCeeCjYY2oGbKf38OL07bZIMlBNmNVSy7qWPti1bbNOFjczpBPXMamXH2jskzSip/9ummOolAB8duTm6L7T7k+/1tYOz6IXQGgPaP5Xu41FBPIhQfwJFugBen5zhLK8rRzpJ1U= ceph-d88002e2-1120-11ee-9b2b-001a4aab830c 2023-06-22T17:20:11.567 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.962+0000 7f3bf2598700 1 Processor -- start 2023-06-22T17:20:11.567 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.962+0000 7f3bf2598700 1 -- start start 2023-06-22T17:20:11.567 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.963+0000 7f3bf2598700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3bec070dc0 0x7f3bec105010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:11.567 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.963+0000 7f3bf2598700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bec105550 con 0x7f3bec070dc0 2023-06-22T17:20:11.567 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.963+0000 7f3bebfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3bec070dc0 0x7f3bec105010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:11.567 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.963+0000 7f3bebfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3bec070dc0 0x7f3bec105010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33552/0 (socket says 172.21.15.39:33552) 2023-06-22T17:20:11.567 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.963+0000 7f3bebfff700 1 -- 172.21.15.39:0/2194560608 learned_addr learned my addr 172.21.15.39:0/2194560608 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:11.568 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.963+0000 7f3bebfff700 1 -- 172.21.15.39:0/2194560608 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bec105690 con 0x7f3bec070dc0 2023-06-22T17:20:11.568 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.963+0000 7f3bebfff700 1 --2- 172.21.15.39:0/2194560608 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3bec070dc0 0x7f3bec105010 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3bd40098e0 tx=0x7f3bd4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=569e17646fac8fff server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:11.568 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.963+0000 7f3beaffd700 1 -- 172.21.15.39:0/2194560608 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3bd4009e80 con 0x7f3bec070dc0 2023-06-22T17:20:11.568 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.963+0000 7f3beaffd700 1 -- 172.21.15.39:0/2194560608 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3bd400bd30 con 0x7f3bec070dc0 2023-06-22T17:20:11.568 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.963+0000 7f3beaffd700 1 -- 172.21.15.39:0/2194560608 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3bd4011da0 con 0x7f3bec070dc0 2023-06-22T17:20:11.569 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.964+0000 7f3bf2598700 1 -- 172.21.15.39:0/2194560608 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3bec070dc0 msgr2=0x7f3bec105010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:11.569 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.964+0000 7f3bf2598700 1 --2- 172.21.15.39:0/2194560608 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3bec070dc0 0x7f3bec105010 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3bd40098e0 tx=0x7f3bd4009250 comp rx=0 tx=0).stop 2023-06-22T17:20:11.569 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.964+0000 7f3bf2598700 1 -- 172.21.15.39:0/2194560608 shutdown_connections 2023-06-22T17:20:11.569 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.964+0000 7f3bf2598700 1 --2- 172.21.15.39:0/2194560608 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3bec070dc0 0x7f3bec105010 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:11.569 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.964+0000 7f3bf2598700 1 -- 172.21.15.39:0/2194560608 >> 172.21.15.39:0/2194560608 conn(0x7f3bec0fca60 msgr2=0x7f3bec0fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:11.569 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.964+0000 7f3bf2598700 1 -- 172.21.15.39:0/2194560608 shutdown_connections 2023-06-22T17:20:11.569 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.964+0000 7f3bf2598700 1 -- 172.21.15.39:0/2194560608 wait complete. 2023-06-22T17:20:11.569 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.965+0000 7f3bf2598700 1 Processor -- start 2023-06-22T17:20:11.570 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.965+0000 7f3bf2598700 1 -- start start 2023-06-22T17:20:11.570 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.965+0000 7f3bf2598700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3bec070dc0 0x7f3bec1110c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:11.570 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.965+0000 7f3bf2598700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bec111600 con 0x7f3bec070dc0 2023-06-22T17:20:11.570 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.965+0000 7f3bebfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3bec070dc0 0x7f3bec1110c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:11.570 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.965+0000 7f3bebfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3bec070dc0 0x7f3bec1110c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33556/0 (socket says 172.21.15.39:33556) 2023-06-22T17:20:11.570 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.965+0000 7f3bebfff700 1 -- 172.21.15.39:0/3663378409 learned_addr learned my addr 172.21.15.39:0/3663378409 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:11.570 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.965+0000 7f3bebfff700 1 -- 172.21.15.39:0/3663378409 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd40094f0 con 0x7f3bec070dc0 2023-06-22T17:20:11.571 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.965+0000 7f3bebfff700 1 --2- 172.21.15.39:0/3663378409 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3bec070dc0 0x7f3bec1110c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3bd4006fd0 tx=0x7f3bd4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:11.571 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.966+0000 7f3be8ff9700 1 -- 172.21.15.39:0/3663378409 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3bd4010b30 con 0x7f3bec070dc0 2023-06-22T17:20:11.571 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.966+0000 7f3bf2598700 1 -- 172.21.15.39:0/3663378409 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bec111800 con 0x7f3bec070dc0 2023-06-22T17:20:11.571 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.966+0000 7f3be8ff9700 1 -- 172.21.15.39:0/3663378409 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3bd4010c90 con 0x7f3bec070dc0 2023-06-22T17:20:11.571 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.966+0000 7f3be8ff9700 1 -- 172.21.15.39:0/3663378409 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3bd4022400 con 0x7f3bec070dc0 2023-06-22T17:20:11.571 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.966+0000 7f3bf2598700 1 -- 172.21.15.39:0/3663378409 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bec111c10 con 0x7f3bec070dc0 2023-06-22T17:20:11.571 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.967+0000 7f3be8ff9700 1 -- 172.21.15.39:0/3663378409 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 8) v1 ==== 43219+0+0 (secure 0 0 0) 0x7f3bd40226c0 con 0x7f3bec070dc0 2023-06-22T17:20:11.571 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.967+0000 7f3bf2598700 1 -- 172.21.15.39:0/3663378409 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bec05f060 con 0x7f3bec070dc0 2023-06-22T17:20:11.572 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.967+0000 7f3be8ff9700 1 --2- 172.21.15.39:0/3663378409 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f3bd8036560 0x7f3bd8038a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:11.572 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.967+0000 7f3be8ff9700 1 -- 172.21.15.39:0/3663378409 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f3bd404abc0 con 0x7f3bec070dc0 2023-06-22T17:20:11.572 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.967+0000 7f3beb7fe700 1 --2- 172.21.15.39:0/3663378409 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f3bd8036560 0x7f3bd8038a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:11.572 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.967+0000 7f3beb7fe700 1 --2- 172.21.15.39:0/3663378409 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f3bd8036560 0x7f3bd8038a10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc009990 tx=0x7f3bdc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:11.572 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:10.972+0000 7f3be8ff9700 1 -- 172.21.15.39:0/3663378409 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f3bd4018350 con 0x7f3bec070dc0 2023-06-22T17:20:11.572 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.170+0000 7f3bf2598700 1 -- 172.21.15.39:0/3663378409 --> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f3bec114670 con 0x7f3bd8036560 2023-06-22T17:20:11.572 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.170+0000 7f3be8ff9700 1 -- 172.21.15.39:0/3663378409 <== mgr.14118 v2:172.21.15.39:6800/1350590065 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f3bec114670 con 0x7f3bd8036560 2023-06-22T17:20:11.573 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.172+0000 7f3bf2598700 1 -- 172.21.15.39:0/3663378409 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f3bd8036560 msgr2=0x7f3bd8038a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:11.573 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.172+0000 7f3bf2598700 1 --2- 172.21.15.39:0/3663378409 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f3bd8036560 0x7f3bd8038a10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc009990 tx=0x7f3bdc006e40 comp rx=0 tx=0).stop 2023-06-22T17:20:11.573 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.172+0000 7f3bf2598700 1 -- 172.21.15.39:0/3663378409 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3bec070dc0 msgr2=0x7f3bec1110c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:11.573 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.172+0000 7f3bf2598700 1 --2- 172.21.15.39:0/3663378409 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3bec070dc0 0x7f3bec1110c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3bd4006fd0 tx=0x7f3bd4009250 comp rx=0 tx=0).stop 2023-06-22T17:20:11.573 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.173+0000 7f3bf2598700 1 -- 172.21.15.39:0/3663378409 shutdown_connections 2023-06-22T17:20:11.573 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.173+0000 7f3bf2598700 1 --2- 172.21.15.39:0/3663378409 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f3bd8036560 0x7f3bd8038a10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:11.573 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.173+0000 7f3bf2598700 1 --2- 172.21.15.39:0/3663378409 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3bec070dc0 0x7f3bec1110c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:11.573 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.173+0000 7f3bf2598700 1 -- 172.21.15.39:0/3663378409 >> 172.21.15.39:0/3663378409 conn(0x7f3bec0fca60 msgr2=0x7f3bec0fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:11.574 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.173+0000 7f3bf2598700 1 -- 172.21.15.39:0/3663378409 shutdown_connections 2023-06-22T17:20:11.574 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.173+0000 7f3bf2598700 1 -- 172.21.15.39:0/3663378409 wait complete. 2023-06-22T17:20:11.574 INFO:teuthology.orchestra.run.smithi039.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-06-22T17:20:11.574 INFO:teuthology.orchestra.run.smithi039.stdout:Adding key to root@localhost authorized_keys... 2023-06-22T17:20:11.574 INFO:teuthology.orchestra.run.smithi039.stdout:Adding host smithi039... 2023-06-22T17:20:11.771 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:11 smithi039 ceph-mon[84805]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:20:12.740 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:12 smithi039 ceph-mon[84805]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi039", "addr": "172.21.15.39", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:20:13.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:13 smithi039 ceph-mon[84805]: Deploying cephadm binary to smithi039 2023-06-22T17:20:14.409 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout Added host 'smithi039' with addr '172.21.15.39' 2023-06-22T17:20:14.410 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.987+0000 7f640599e700 1 Processor -- start 2023-06-22T17:20:14.410 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.987+0000 7f640599e700 1 -- start start 2023-06-22T17:20:14.410 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.988+0000 7f640599e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6400101160 0x7f6400101570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:14.410 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.988+0000 7f640599e700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6400101ab0 con 0x7f6400101160 2023-06-22T17:20:14.410 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.988+0000 7f640499c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6400101160 0x7f6400101570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:14.410 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.988+0000 7f640499c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6400101160 0x7f6400101570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33564/0 (socket says 172.21.15.39:33564) 2023-06-22T17:20:14.411 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.988+0000 7f640499c700 1 -- 172.21.15.39:0/1098086201 learned_addr learned my addr 172.21.15.39:0/1098086201 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:14.411 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.988+0000 7f640499c700 1 -- 172.21.15.39:0/1098086201 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6400101bf0 con 0x7f6400101160 2023-06-22T17:20:14.411 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.988+0000 7f640499c700 1 --2- 172.21.15.39:0/1098086201 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6400101160 0x7f6400101570 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f63f40098e0 tx=0x7f63f4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd72276e4d3054ab server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:14.411 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.989+0000 7f63ff7fe700 1 -- 172.21.15.39:0/1098086201 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f63f4009e80 con 0x7f6400101160 2023-06-22T17:20:14.411 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.989+0000 7f63ff7fe700 1 -- 172.21.15.39:0/1098086201 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f63f400bd30 con 0x7f6400101160 2023-06-22T17:20:14.411 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.989+0000 7f63ff7fe700 1 -- 172.21.15.39:0/1098086201 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f63f4011da0 con 0x7f6400101160 2023-06-22T17:20:14.411 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.989+0000 7f640599e700 1 -- 172.21.15.39:0/1098086201 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6400101160 msgr2=0x7f6400101570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:14.412 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.989+0000 7f640599e700 1 --2- 172.21.15.39:0/1098086201 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6400101160 0x7f6400101570 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f63f40098e0 tx=0x7f63f4009250 comp rx=0 tx=0).stop 2023-06-22T17:20:14.412 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.989+0000 7f640599e700 1 -- 172.21.15.39:0/1098086201 shutdown_connections 2023-06-22T17:20:14.412 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.989+0000 7f640599e700 1 --2- 172.21.15.39:0/1098086201 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6400101160 0x7f6400101570 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:14.412 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.989+0000 7f640599e700 1 -- 172.21.15.39:0/1098086201 >> 172.21.15.39:0/1098086201 conn(0x7f64000fc8e0 msgr2=0x7f64000fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:14.412 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.989+0000 7f640599e700 1 -- 172.21.15.39:0/1098086201 shutdown_connections 2023-06-22T17:20:14.412 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.989+0000 7f640599e700 1 -- 172.21.15.39:0/1098086201 wait complete. 2023-06-22T17:20:14.413 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.990+0000 7f640599e700 1 Processor -- start 2023-06-22T17:20:14.413 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.990+0000 7f640599e700 1 -- start start 2023-06-22T17:20:14.413 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.990+0000 7f640599e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6400101160 0x7f640010ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:14.413 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.990+0000 7f640599e700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640010f270 con 0x7f6400101160 2023-06-22T17:20:14.413 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.990+0000 7f640499c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6400101160 0x7f640010ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:14.413 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.990+0000 7f640499c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6400101160 0x7f640010ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33570/0 (socket says 172.21.15.39:33570) 2023-06-22T17:20:14.413 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.990+0000 7f640499c700 1 -- 172.21.15.39:0/270691403 learned_addr learned my addr 172.21.15.39:0/270691403 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:14.413 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.990+0000 7f640499c700 1 -- 172.21.15.39:0/270691403 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f40094f0 con 0x7f6400101160 2023-06-22T17:20:14.414 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.991+0000 7f640499c700 1 --2- 172.21.15.39:0/270691403 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6400101160 0x7f640010ed30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f63f4006b00 tx=0x7f63f4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:14.414 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.991+0000 7f63fd7fa700 1 -- 172.21.15.39:0/270691403 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f63f4010b30 con 0x7f6400101160 2023-06-22T17:20:14.414 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.991+0000 7f640599e700 1 -- 172.21.15.39:0/270691403 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f640010f470 con 0x7f6400101160 2023-06-22T17:20:14.414 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.991+0000 7f63fd7fa700 1 -- 172.21.15.39:0/270691403 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f63f4010c90 con 0x7f6400101160 2023-06-22T17:20:14.414 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.991+0000 7f63fd7fa700 1 -- 172.21.15.39:0/270691403 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f63f4022400 con 0x7f6400101160 2023-06-22T17:20:14.414 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.991+0000 7f640599e700 1 -- 172.21.15.39:0/270691403 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f640010f880 con 0x7f6400101160 2023-06-22T17:20:14.414 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.992+0000 7f63fd7fa700 1 -- 172.21.15.39:0/270691403 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 8) v1 ==== 43219+0+0 (secure 0 0 0) 0x7f63f401a430 con 0x7f6400101160 2023-06-22T17:20:14.415 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.992+0000 7f63fd7fa700 1 --2- 172.21.15.39:0/270691403 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f63f00387c0 0x7f63f003ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:14.415 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.992+0000 7f63fd7fa700 1 -- 172.21.15.39:0/270691403 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f63f404a840 con 0x7f6400101160 2023-06-22T17:20:14.415 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.992+0000 7f640599e700 1 -- 172.21.15.39:0/270691403 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f640005f060 con 0x7f6400101160 2023-06-22T17:20:14.415 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.992+0000 7f63fffff700 1 --2- 172.21.15.39:0/270691403 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f63f00387c0 0x7f63f003ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:14.415 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.992+0000 7f63fffff700 1 --2- 172.21.15.39:0/270691403 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f63f00387c0 0x7f63f003ac70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f63ec009990 tx=0x7f63ec006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:14.415 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:11.997+0000 7f63fd7fa700 1 -- 172.21.15.39:0/270691403 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f63f402ab80 con 0x7f6400101160 2023-06-22T17:20:14.415 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:12.194+0000 7f640599e700 1 -- 172.21.15.39:0/270691403 --> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi039", "addr": "172.21.15.39", "target": ["mon-mgr", ""]}) v1 -- 0x7f640002b0d0 con 0x7f63f00387c0 2023-06-22T17:20:14.416 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.037+0000 7f63fd7fa700 1 -- 172.21.15.39:0/270691403 <== mgr.14118 v2:172.21.15.39:6800/1350590065 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f640002b0d0 con 0x7f63f00387c0 2023-06-22T17:20:14.416 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.040+0000 7f640599e700 1 -- 172.21.15.39:0/270691403 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f63f00387c0 msgr2=0x7f63f003ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:14.416 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.040+0000 7f640599e700 1 --2- 172.21.15.39:0/270691403 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f63f00387c0 0x7f63f003ac70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f63ec009990 tx=0x7f63ec006e40 comp rx=0 tx=0).stop 2023-06-22T17:20:14.416 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.040+0000 7f640599e700 1 -- 172.21.15.39:0/270691403 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6400101160 msgr2=0x7f640010ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:14.416 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.040+0000 7f640599e700 1 --2- 172.21.15.39:0/270691403 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6400101160 0x7f640010ed30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f63f4006b00 tx=0x7f63f4009250 comp rx=0 tx=0).stop 2023-06-22T17:20:14.416 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.040+0000 7f640599e700 1 -- 172.21.15.39:0/270691403 shutdown_connections 2023-06-22T17:20:14.416 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.040+0000 7f640599e700 1 --2- 172.21.15.39:0/270691403 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f63f00387c0 0x7f63f003ac70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:14.416 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.040+0000 7f640599e700 1 --2- 172.21.15.39:0/270691403 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6400101160 0x7f640010ed30 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:14.417 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.040+0000 7f640599e700 1 -- 172.21.15.39:0/270691403 >> 172.21.15.39:0/270691403 conn(0x7f64000fc8e0 msgr2=0x7f64000fece0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:14.417 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.040+0000 7f640599e700 1 -- 172.21.15.39:0/270691403 shutdown_connections 2023-06-22T17:20:14.417 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.040+0000 7f640599e700 1 -- 172.21.15.39:0/270691403 wait complete. 2023-06-22T17:20:14.417 INFO:teuthology.orchestra.run.smithi039.stdout:Deploying unmanaged mon service... 2023-06-22T17:20:15.292 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:15 smithi039 ceph-mon[84805]: from='mgr.14118 172.21.15.39:0/1180832241' entity='mgr.x' 2023-06-22T17:20:15.293 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:15 smithi039 ceph-mon[84805]: Added host smithi039 2023-06-22T17:20:15.293 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:15 smithi039 ceph-mon[84805]: from='mgr.14118 172.21.15.39:0/1180832241' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:20:15.547 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-06-22T17:20:15.548 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.884+0000 7f72352a5700 1 Processor -- start 2023-06-22T17:20:15.548 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.884+0000 7f72352a5700 1 -- start start 2023-06-22T17:20:15.548 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.884+0000 7f72352a5700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7230101300 0x7f7230101710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:15.548 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.884+0000 7f72352a5700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7230101c50 con 0x7f7230101300 2023-06-22T17:20:15.548 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.885+0000 7f722effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7230101300 0x7f7230101710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:15.548 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.885+0000 7f722effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7230101300 0x7f7230101710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33586/0 (socket says 172.21.15.39:33586) 2023-06-22T17:20:15.548 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.885+0000 7f722effd700 1 -- 172.21.15.39:0/3468404211 learned_addr learned my addr 172.21.15.39:0/3468404211 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:15.549 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.885+0000 7f722effd700 1 -- 172.21.15.39:0/3468404211 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7230101d90 con 0x7f7230101300 2023-06-22T17:20:15.549 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.885+0000 7f722effd700 1 --2- 172.21.15.39:0/3468404211 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7230101300 0x7f7230101710 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f722800bb80 tx=0x7f722800b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae0098841a3a61d4 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:15.549 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.885+0000 7f722dffb700 1 -- 172.21.15.39:0/3468404211 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f722800dba0 con 0x7f7230101300 2023-06-22T17:20:15.549 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.885+0000 7f722dffb700 1 -- 172.21.15.39:0/3468404211 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f722800dd00 con 0x7f7230101300 2023-06-22T17:20:15.549 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.885+0000 7f722dffb700 1 -- 172.21.15.39:0/3468404211 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f72280187f0 con 0x7f7230101300 2023-06-22T17:20:15.549 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.886+0000 7f72352a5700 1 -- 172.21.15.39:0/3468404211 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7230101300 msgr2=0x7f7230101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:15.549 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.886+0000 7f72352a5700 1 --2- 172.21.15.39:0/3468404211 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7230101300 0x7f7230101710 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f722800bb80 tx=0x7f722800b4f0 comp rx=0 tx=0).stop 2023-06-22T17:20:15.549 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.886+0000 7f72352a5700 1 -- 172.21.15.39:0/3468404211 shutdown_connections 2023-06-22T17:20:15.550 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.886+0000 7f72352a5700 1 --2- 172.21.15.39:0/3468404211 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7230101300 0x7f7230101710 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:15.550 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.886+0000 7f72352a5700 1 -- 172.21.15.39:0/3468404211 >> 172.21.15.39:0/3468404211 conn(0x7f72300fca60 msgr2=0x7f72300fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:15.550 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.886+0000 7f72352a5700 1 -- 172.21.15.39:0/3468404211 shutdown_connections 2023-06-22T17:20:15.550 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.886+0000 7f72352a5700 1 -- 172.21.15.39:0/3468404211 wait complete. 2023-06-22T17:20:15.550 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.887+0000 7f72352a5700 1 Processor -- start 2023-06-22T17:20:15.550 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.887+0000 7f72352a5700 1 -- start start 2023-06-22T17:20:15.550 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.887+0000 7f72352a5700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7230101300 0x7f723010ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:15.551 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.887+0000 7f72352a5700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723010f360 con 0x7f7230101300 2023-06-22T17:20:15.551 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.887+0000 7f722effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7230101300 0x7f723010ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:15.551 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.887+0000 7f722effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7230101300 0x7f723010ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33600/0 (socket says 172.21.15.39:33600) 2023-06-22T17:20:15.551 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.887+0000 7f722effd700 1 -- 172.21.15.39:0/45494823 learned_addr learned my addr 172.21.15.39:0/45494823 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:15.551 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.887+0000 7f722effd700 1 -- 172.21.15.39:0/45494823 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f722800b790 con 0x7f7230101300 2023-06-22T17:20:15.551 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.888+0000 7f722effd700 1 --2- 172.21.15.39:0/45494823 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7230101300 0x7f723010ee20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7228003080 tx=0x7f722800b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:15.551 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.888+0000 7f72277fe700 1 -- 172.21.15.39:0/45494823 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7228027070 con 0x7f7230101300 2023-06-22T17:20:15.551 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.888+0000 7f72277fe700 1 -- 172.21.15.39:0/45494823 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f722801d050 con 0x7f7230101300 2023-06-22T17:20:15.552 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.888+0000 7f72352a5700 1 -- 172.21.15.39:0/45494823 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f723010f560 con 0x7f7230101300 2023-06-22T17:20:15.552 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.888+0000 7f72277fe700 1 -- 172.21.15.39:0/45494823 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f72280223d0 con 0x7f7230101300 2023-06-22T17:20:15.552 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.888+0000 7f72352a5700 1 -- 172.21.15.39:0/45494823 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f723010f970 con 0x7f7230101300 2023-06-22T17:20:15.552 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.889+0000 7f72352a5700 1 -- 172.21.15.39:0/45494823 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72301087e0 con 0x7f7230101300 2023-06-22T17:20:15.552 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.889+0000 7f72277fe700 1 -- 172.21.15.39:0/45494823 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 8) v1 ==== 43219+0+0 (secure 0 0 0) 0x7f72280108d0 con 0x7f7230101300 2023-06-22T17:20:15.552 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.889+0000 7f72277fe700 1 --2- 172.21.15.39:0/45494823 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f721c036560 0x7f721c038a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:15.552 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.889+0000 7f72277fe700 1 -- 172.21.15.39:0/45494823 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f7228014070 con 0x7f7230101300 2023-06-22T17:20:15.553 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.889+0000 7f722e7fc700 1 --2- 172.21.15.39:0/45494823 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f721c036560 0x7f721c038a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:15.553 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.890+0000 7f722e7fc700 1 --2- 172.21.15.39:0/45494823 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f721c036560 0x7f721c038a10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7218009990 tx=0x7f7218006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:15.553 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:14.894+0000 7f72277fe700 1 -- 172.21.15.39:0/45494823 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f7228023470 con 0x7f7230101300 2023-06-22T17:20:15.553 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:15.102+0000 7f72352a5700 1 -- 172.21.15.39:0/45494823 --> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f723002b1b0 con 0x7f721c036560 2023-06-22T17:20:15.553 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:15.107+0000 7f72277fe700 1 -- 172.21.15.39:0/45494823 <== mgr.14118 v2:172.21.15.39:6800/1350590065 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f723002b1b0 con 0x7f721c036560 2023-06-22T17:20:15.553 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:15.109+0000 7f72352a5700 1 -- 172.21.15.39:0/45494823 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f721c036560 msgr2=0x7f721c038a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:15.553 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:15.109+0000 7f72352a5700 1 --2- 172.21.15.39:0/45494823 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f721c036560 0x7f721c038a10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7218009990 tx=0x7f7218006e40 comp rx=0 tx=0).stop 2023-06-22T17:20:15.553 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:15.109+0000 7f72352a5700 1 -- 172.21.15.39:0/45494823 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7230101300 msgr2=0x7f723010ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:15.554 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:15.109+0000 7f72352a5700 1 --2- 172.21.15.39:0/45494823 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7230101300 0x7f723010ee20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7228003080 tx=0x7f722800b4f0 comp rx=0 tx=0).stop 2023-06-22T17:20:15.554 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:15.109+0000 7f72352a5700 1 -- 172.21.15.39:0/45494823 shutdown_connections 2023-06-22T17:20:15.554 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:15.109+0000 7f72352a5700 1 --2- 172.21.15.39:0/45494823 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f721c036560 0x7f721c038a10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:15.554 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:15.109+0000 7f72352a5700 1 --2- 172.21.15.39:0/45494823 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7230101300 0x7f723010ee20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:15.554 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:15.109+0000 7f72352a5700 1 -- 172.21.15.39:0/45494823 >> 172.21.15.39:0/45494823 conn(0x7f72300fca60 msgr2=0x7f72300fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:15.554 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:15.109+0000 7f72352a5700 1 -- 172.21.15.39:0/45494823 shutdown_connections 2023-06-22T17:20:15.554 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:15.109+0000 7f72352a5700 1 -- 172.21.15.39:0/45494823 wait complete. 2023-06-22T17:20:15.554 INFO:teuthology.orchestra.run.smithi039.stdout:Deploying unmanaged mgr service... 2023-06-22T17:20:16.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:16 smithi039 ceph-mon[84805]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:20:16.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:16 smithi039 ceph-mon[84805]: Saving service mon spec with placement count:5 2023-06-22T17:20:16.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:16 smithi039 ceph-mon[84805]: from='mgr.14118 172.21.15.39:0/1180832241' entity='mgr.x' 2023-06-22T17:20:16.689 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-06-22T17:20:16.690 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.055+0000 7f0172b19700 1 Processor -- start 2023-06-22T17:20:16.690 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.055+0000 7f0172b19700 1 -- start start 2023-06-22T17:20:16.690 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.055+0000 7f0172b19700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f016c103a20 0x7f016c103e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:16.690 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.055+0000 7f0172b19700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016c104370 con 0x7f016c103a20 2023-06-22T17:20:16.691 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.055+0000 7f01708b5700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f016c103a20 0x7f016c103e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:16.691 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.055+0000 7f01708b5700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f016c103a20 0x7f016c103e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35258/0 (socket says 172.21.15.39:35258) 2023-06-22T17:20:16.691 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.055+0000 7f01708b5700 1 -- 172.21.15.39:0/1052672954 learned_addr learned my addr 172.21.15.39:0/1052672954 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:16.691 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.056+0000 7f01708b5700 1 -- 172.21.15.39:0/1052672954 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f016c1044b0 con 0x7f016c103a20 2023-06-22T17:20:16.691 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.056+0000 7f01708b5700 1 --2- 172.21.15.39:0/1052672954 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f016c103a20 0x7f016c103e30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f01600098e0 tx=0x7f0160009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8708701cd9d153ca server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:16.691 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.056+0000 7f016b7fe700 1 -- 172.21.15.39:0/1052672954 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0160009e80 con 0x7f016c103a20 2023-06-22T17:20:16.691 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.056+0000 7f016b7fe700 1 -- 172.21.15.39:0/1052672954 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f016000bd30 con 0x7f016c103a20 2023-06-22T17:20:16.691 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.056+0000 7f016b7fe700 1 -- 172.21.15.39:0/1052672954 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0160011da0 con 0x7f016c103a20 2023-06-22T17:20:16.692 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.057+0000 7f0172b19700 1 -- 172.21.15.39:0/1052672954 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f016c103a20 msgr2=0x7f016c103e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:16.692 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.057+0000 7f0172b19700 1 --2- 172.21.15.39:0/1052672954 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f016c103a20 0x7f016c103e30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f01600098e0 tx=0x7f0160009250 comp rx=0 tx=0).stop 2023-06-22T17:20:16.692 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.057+0000 7f0172b19700 1 -- 172.21.15.39:0/1052672954 shutdown_connections 2023-06-22T17:20:16.692 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.057+0000 7f0172b19700 1 --2- 172.21.15.39:0/1052672954 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f016c103a20 0x7f016c103e30 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:16.692 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.057+0000 7f0172b19700 1 -- 172.21.15.39:0/1052672954 >> 172.21.15.39:0/1052672954 conn(0x7f016c0fc1e0 msgr2=0x7f016c0fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:16.692 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.057+0000 7f0172b19700 1 -- 172.21.15.39:0/1052672954 shutdown_connections 2023-06-22T17:20:16.692 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.057+0000 7f0172b19700 1 -- 172.21.15.39:0/1052672954 wait complete. 2023-06-22T17:20:16.693 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.057+0000 7f0172b19700 1 Processor -- start 2023-06-22T17:20:16.693 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.057+0000 7f0172b19700 1 -- start start 2023-06-22T17:20:16.693 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.058+0000 7f0172b19700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f016c103a20 0x7f016c111030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:16.693 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.058+0000 7f0172b19700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016c111570 con 0x7f016c103a20 2023-06-22T17:20:16.693 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.058+0000 7f01708b5700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f016c103a20 0x7f016c111030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:16.693 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.058+0000 7f01708b5700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f016c103a20 0x7f016c111030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35264/0 (socket says 172.21.15.39:35264) 2023-06-22T17:20:16.693 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.058+0000 7f01708b5700 1 -- 172.21.15.39:0/868622923 learned_addr learned my addr 172.21.15.39:0/868622923 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:16.693 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.058+0000 7f01708b5700 1 -- 172.21.15.39:0/868622923 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01600094f0 con 0x7f016c103a20 2023-06-22T17:20:16.694 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.058+0000 7f01708b5700 1 --2- 172.21.15.39:0/868622923 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f016c103a20 0x7f016c111030 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0160000c00 tx=0x7f0160009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:16.694 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.058+0000 7f01697fa700 1 -- 172.21.15.39:0/868622923 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0160010b30 con 0x7f016c103a20 2023-06-22T17:20:16.694 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.059+0000 7f01697fa700 1 -- 172.21.15.39:0/868622923 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0160010c90 con 0x7f016c103a20 2023-06-22T17:20:16.694 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.059+0000 7f0172b19700 1 -- 172.21.15.39:0/868622923 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f016c111770 con 0x7f016c103a20 2023-06-22T17:20:16.694 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.059+0000 7f01697fa700 1 -- 172.21.15.39:0/868622923 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0160022400 con 0x7f016c103a20 2023-06-22T17:20:16.694 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.059+0000 7f0172b19700 1 -- 172.21.15.39:0/868622923 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f016c111b80 con 0x7f016c103a20 2023-06-22T17:20:16.694 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.059+0000 7f0172b19700 1 -- 172.21.15.39:0/868622923 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f016c05f060 con 0x7f016c103a20 2023-06-22T17:20:16.694 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.060+0000 7f01697fa700 1 -- 172.21.15.39:0/868622923 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 8) v1 ==== 43219+0+0 (secure 0 0 0) 0x7f016001a430 con 0x7f016c103a20 2023-06-22T17:20:16.695 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.060+0000 7f01697fa700 1 --2- 172.21.15.39:0/868622923 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f015c036560 0x7f015c038a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:16.695 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.060+0000 7f01697fa700 1 -- 172.21.15.39:0/868622923 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f016004ac70 con 0x7f016c103a20 2023-06-22T17:20:16.695 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.060+0000 7f016bfff700 1 --2- 172.21.15.39:0/868622923 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f015c036560 0x7f015c038a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:16.695 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.061+0000 7f016bfff700 1 --2- 172.21.15.39:0/868622923 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f015c036560 0x7f015c038a10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0158009990 tx=0x7f0158006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:16.695 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.065+0000 7f01697fa700 1 -- 172.21.15.39:0/868622923 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f0160021d30 con 0x7f016c103a20 2023-06-22T17:20:16.695 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.269+0000 7f0172b19700 1 -- 172.21.15.39:0/868622923 --> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f016c1145e0 con 0x7f015c036560 2023-06-22T17:20:16.695 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.279+0000 7f01697fa700 1 -- 172.21.15.39:0/868622923 <== mgr.14118 v2:172.21.15.39:6800/1350590065 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f016c1145e0 con 0x7f015c036560 2023-06-22T17:20:16.696 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.281+0000 7f0172b19700 1 -- 172.21.15.39:0/868622923 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f015c036560 msgr2=0x7f015c038a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:16.696 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.281+0000 7f0172b19700 1 --2- 172.21.15.39:0/868622923 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f015c036560 0x7f015c038a10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0158009990 tx=0x7f0158006e40 comp rx=0 tx=0).stop 2023-06-22T17:20:16.696 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.281+0000 7f0172b19700 1 -- 172.21.15.39:0/868622923 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f016c103a20 msgr2=0x7f016c111030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:16.696 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.281+0000 7f0172b19700 1 --2- 172.21.15.39:0/868622923 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f016c103a20 0x7f016c111030 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0160000c00 tx=0x7f0160009250 comp rx=0 tx=0).stop 2023-06-22T17:20:16.696 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.281+0000 7f0172b19700 1 -- 172.21.15.39:0/868622923 shutdown_connections 2023-06-22T17:20:16.696 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.281+0000 7f0172b19700 1 --2- 172.21.15.39:0/868622923 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f015c036560 0x7f015c038a10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:16.696 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.281+0000 7f0172b19700 1 --2- 172.21.15.39:0/868622923 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f016c103a20 0x7f016c111030 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:16.696 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.281+0000 7f0172b19700 1 -- 172.21.15.39:0/868622923 >> 172.21.15.39:0/868622923 conn(0x7f016c0fc1e0 msgr2=0x7f016c0fcf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:16.697 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.281+0000 7f0172b19700 1 -- 172.21.15.39:0/868622923 shutdown_connections 2023-06-22T17:20:16.697 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:16.281+0000 7f0172b19700 1 -- 172.21.15.39:0/868622923 wait complete. 2023-06-22T17:20:17.541 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:17 smithi039 ceph-mon[84805]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:20:17.542 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:17 smithi039 ceph-mon[84805]: Saving service mgr spec with placement count:2 2023-06-22T17:20:17.542 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:17 smithi039 ceph-mon[84805]: from='mgr.14118 172.21.15.39:0/1180832241' entity='mgr.x' 2023-06-22T17:20:17.542 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:17 smithi039 ceph-mon[84805]: from='mgr.14118 172.21.15.39:0/1180832241' entity='mgr.x' 2023-06-22T17:20:17.542 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:17 smithi039 ceph-mon[84805]: from='mgr.14118 172.21.15.39:0/1180832241' entity='mgr.x' 2023-06-22T17:20:17.729 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.165+0000 7f93afc7c700 1 Processor -- start 2023-06-22T17:20:17.729 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.165+0000 7f93afc7c700 1 -- start start 2023-06-22T17:20:17.729 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.165+0000 7f93afc7c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93a8101320 0x7f93a8101730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:17.729 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.165+0000 7f93afc7c700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a8101c70 con 0x7f93a8101320 2023-06-22T17:20:17.730 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.165+0000 7f93ada18700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93a8101320 0x7f93a8101730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:17.730 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.165+0000 7f93ada18700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93a8101320 0x7f93a8101730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35274/0 (socket says 172.21.15.39:35274) 2023-06-22T17:20:17.730 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.165+0000 7f93ada18700 1 -- 172.21.15.39:0/1038746299 learned_addr learned my addr 172.21.15.39:0/1038746299 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:17.730 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.166+0000 7f93ada18700 1 -- 172.21.15.39:0/1038746299 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a8101db0 con 0x7f93a8101320 2023-06-22T17:20:17.730 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.166+0000 7f93ada18700 1 --2- 172.21.15.39:0/1038746299 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93a8101320 0x7f93a8101730 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f93a400bb80 tx=0x7f93a400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d1b62c10d74005b server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:17.730 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.166+0000 7f93aca16700 1 -- 172.21.15.39:0/1038746299 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f93a400dba0 con 0x7f93a8101320 2023-06-22T17:20:17.730 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.166+0000 7f93aca16700 1 -- 172.21.15.39:0/1038746299 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f93a400dd00 con 0x7f93a8101320 2023-06-22T17:20:17.730 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.166+0000 7f93aca16700 1 -- 172.21.15.39:0/1038746299 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f93a4018780 con 0x7f93a8101320 2023-06-22T17:20:17.731 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.166+0000 7f93afc7c700 1 -- 172.21.15.39:0/1038746299 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93a8101320 msgr2=0x7f93a8101730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:17.731 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.167+0000 7f93afc7c700 1 --2- 172.21.15.39:0/1038746299 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93a8101320 0x7f93a8101730 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f93a400bb80 tx=0x7f93a400b4f0 comp rx=0 tx=0).stop 2023-06-22T17:20:17.731 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.167+0000 7f93afc7c700 1 -- 172.21.15.39:0/1038746299 shutdown_connections 2023-06-22T17:20:17.731 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.167+0000 7f93afc7c700 1 --2- 172.21.15.39:0/1038746299 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93a8101320 0x7f93a8101730 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:17.731 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.167+0000 7f93afc7c700 1 -- 172.21.15.39:0/1038746299 >> 172.21.15.39:0/1038746299 conn(0x7f93a80fcac0 msgr2=0x7f93a80feed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:17.731 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.167+0000 7f93afc7c700 1 -- 172.21.15.39:0/1038746299 shutdown_connections 2023-06-22T17:20:17.731 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.167+0000 7f93afc7c700 1 -- 172.21.15.39:0/1038746299 wait complete. 2023-06-22T17:20:17.731 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.167+0000 7f93afc7c700 1 Processor -- start 2023-06-22T17:20:17.732 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.167+0000 7f93afc7c700 1 -- start start 2023-06-22T17:20:17.732 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.167+0000 7f93afc7c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93a8101320 0x7f93a810eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:17.732 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.167+0000 7f93afc7c700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a810f3f0 con 0x7f93a8101320 2023-06-22T17:20:17.732 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.167+0000 7f93ada18700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93a8101320 0x7f93a810eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:17.732 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.168+0000 7f93ada18700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93a8101320 0x7f93a810eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35290/0 (socket says 172.21.15.39:35290) 2023-06-22T17:20:17.732 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.168+0000 7f93ada18700 1 -- 172.21.15.39:0/3008971181 learned_addr learned my addr 172.21.15.39:0/3008971181 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:17.732 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.168+0000 7f93ada18700 1 -- 172.21.15.39:0/3008971181 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a400b790 con 0x7f93a8101320 2023-06-22T17:20:17.733 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.168+0000 7f93ada18700 1 --2- 172.21.15.39:0/3008971181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93a8101320 0x7f93a810eeb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f93a4007010 tx=0x7f93a400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:17.733 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.168+0000 7f939e7fc700 1 -- 172.21.15.39:0/3008971181 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f93a4010a30 con 0x7f93a8101320 2023-06-22T17:20:17.733 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.168+0000 7f939e7fc700 1 -- 172.21.15.39:0/3008971181 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f93a401d040 con 0x7f93a8101320 2023-06-22T17:20:17.733 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.168+0000 7f939e7fc700 1 -- 172.21.15.39:0/3008971181 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f93a4020530 con 0x7f93a8101320 2023-06-22T17:20:17.733 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.168+0000 7f93afc7c700 1 -- 172.21.15.39:0/3008971181 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93a810f5f0 con 0x7f93a8101320 2023-06-22T17:20:17.733 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.168+0000 7f93afc7c700 1 -- 172.21.15.39:0/3008971181 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93a810fa00 con 0x7f93a8101320 2023-06-22T17:20:17.733 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.169+0000 7f939e7fc700 1 -- 172.21.15.39:0/3008971181 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 8) v1 ==== 43219+0+0 (secure 0 0 0) 0x7f93a40206d0 con 0x7f93a8101320 2023-06-22T17:20:17.733 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.169+0000 7f93afc7c700 1 -- 172.21.15.39:0/3008971181 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f938c0052e0 con 0x7f93a8101320 2023-06-22T17:20:17.734 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.169+0000 7f939e7fc700 1 --2- 172.21.15.39:0/3008971181 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f93940365c0 0x7f9394038a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:17.734 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.169+0000 7f939e7fc700 1 -- 172.21.15.39:0/3008971181 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f93a4014070 con 0x7f93a8101320 2023-06-22T17:20:17.734 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.169+0000 7f93ad217700 1 --2- 172.21.15.39:0/3008971181 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f93940365c0 0x7f9394038a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:17.734 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.170+0000 7f93ad217700 1 --2- 172.21.15.39:0/3008971181 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f93940365c0 0x7f9394038a70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9398009990 tx=0x7f9398006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:17.734 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.174+0000 7f939e7fc700 1 -- 172.21.15.39:0/3008971181 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f93a400e950 con 0x7f93a8101320 2023-06-22T17:20:17.734 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.373+0000 7f93afc7c700 1 -- 172.21.15.39:0/3008971181 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f938c005590 con 0x7f93a8101320 2023-06-22T17:20:17.734 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.381+0000 7f939e7fc700 1 -- 172.21.15.39:0/3008971181 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f93a4028030 con 0x7f93a8101320 2023-06-22T17:20:17.734 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.384+0000 7f93afc7c700 1 -- 172.21.15.39:0/3008971181 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f93940365c0 msgr2=0x7f9394038a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:17.735 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.384+0000 7f93afc7c700 1 --2- 172.21.15.39:0/3008971181 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f93940365c0 0x7f9394038a70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9398009990 tx=0x7f9398006e40 comp rx=0 tx=0).stop 2023-06-22T17:20:17.735 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.384+0000 7f93afc7c700 1 -- 172.21.15.39:0/3008971181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93a8101320 msgr2=0x7f93a810eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:17.735 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.384+0000 7f93afc7c700 1 --2- 172.21.15.39:0/3008971181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93a8101320 0x7f93a810eeb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f93a4007010 tx=0x7f93a400b4f0 comp rx=0 tx=0).stop 2023-06-22T17:20:17.735 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.384+0000 7f93afc7c700 1 -- 172.21.15.39:0/3008971181 shutdown_connections 2023-06-22T17:20:17.735 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.384+0000 7f93afc7c700 1 --2- 172.21.15.39:0/3008971181 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f93940365c0 0x7f9394038a70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:17.735 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.384+0000 7f93afc7c700 1 --2- 172.21.15.39:0/3008971181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93a8101320 0x7f93a810eeb0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:17.735 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.384+0000 7f93afc7c700 1 -- 172.21.15.39:0/3008971181 >> 172.21.15.39:0/3008971181 conn(0x7f93a80fcac0 msgr2=0x7f93a80fee80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:17.735 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.384+0000 7f93afc7c700 1 -- 172.21.15.39:0/3008971181 shutdown_connections 2023-06-22T17:20:17.736 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:17.384+0000 7f93afc7c700 1 -- 172.21.15.39:0/3008971181 wait complete. 2023-06-22T17:20:18.666 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:18 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/3008971181' entity='client.admin' 2023-06-22T17:20:18.867 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.181+0000 7f05274a7700 1 Processor -- start 2023-06-22T17:20:18.868 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.181+0000 7f05274a7700 1 -- start start 2023-06-22T17:20:18.868 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.181+0000 7f05274a7700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0520070dc0 0x7f0520105010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:18.868 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.181+0000 7f05274a7700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0520105550 con 0x7f0520070dc0 2023-06-22T17:20:18.868 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.181+0000 7f0525243700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0520070dc0 0x7f0520105010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:18.868 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.181+0000 7f0525243700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0520070dc0 0x7f0520105010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35292/0 (socket says 172.21.15.39:35292) 2023-06-22T17:20:18.868 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.181+0000 7f0525243700 1 -- 172.21.15.39:0/2058763868 learned_addr learned my addr 172.21.15.39:0/2058763868 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:18.869 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.182+0000 7f0525243700 1 -- 172.21.15.39:0/2058763868 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0520105690 con 0x7f0520070dc0 2023-06-22T17:20:18.869 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.182+0000 7f0525243700 1 --2- 172.21.15.39:0/2058763868 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0520070dc0 0x7f0520105010 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f051c0098e0 tx=0x7f051c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f9088309578d332 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:18.869 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.182+0000 7f0517fff700 1 -- 172.21.15.39:0/2058763868 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f051c009e80 con 0x7f0520070dc0 2023-06-22T17:20:18.869 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.182+0000 7f0517fff700 1 -- 172.21.15.39:0/2058763868 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f051c00bd40 con 0x7f0520070dc0 2023-06-22T17:20:18.869 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.182+0000 7f0517fff700 1 -- 172.21.15.39:0/2058763868 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f051c011d10 con 0x7f0520070dc0 2023-06-22T17:20:18.869 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.183+0000 7f05274a7700 1 -- 172.21.15.39:0/2058763868 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0520070dc0 msgr2=0x7f0520105010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:18.870 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.183+0000 7f05274a7700 1 --2- 172.21.15.39:0/2058763868 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0520070dc0 0x7f0520105010 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f051c0098e0 tx=0x7f051c009250 comp rx=0 tx=0).stop 2023-06-22T17:20:18.870 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.183+0000 7f05274a7700 1 -- 172.21.15.39:0/2058763868 shutdown_connections 2023-06-22T17:20:18.870 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.183+0000 7f05274a7700 1 --2- 172.21.15.39:0/2058763868 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0520070dc0 0x7f0520105010 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:18.870 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.183+0000 7f05274a7700 1 -- 172.21.15.39:0/2058763868 >> 172.21.15.39:0/2058763868 conn(0x7f05200fca60 msgr2=0x7f05200fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:18.870 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.183+0000 7f05274a7700 1 -- 172.21.15.39:0/2058763868 shutdown_connections 2023-06-22T17:20:18.870 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.183+0000 7f05274a7700 1 -- 172.21.15.39:0/2058763868 wait complete. 2023-06-22T17:20:18.870 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.183+0000 7f05274a7700 1 Processor -- start 2023-06-22T17:20:18.871 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.183+0000 7f05274a7700 1 -- start start 2023-06-22T17:20:18.871 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.183+0000 7f05274a7700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0520070dc0 0x7f05201110e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:18.871 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.183+0000 7f05274a7700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0520111620 con 0x7f0520070dc0 2023-06-22T17:20:18.871 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.184+0000 7f0525243700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0520070dc0 0x7f05201110e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:18.871 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.184+0000 7f0525243700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0520070dc0 0x7f05201110e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35306/0 (socket says 172.21.15.39:35306) 2023-06-22T17:20:18.871 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.184+0000 7f0525243700 1 -- 172.21.15.39:0/959588581 learned_addr learned my addr 172.21.15.39:0/959588581 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:18.871 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.184+0000 7f0525243700 1 -- 172.21.15.39:0/959588581 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f051c0094f0 con 0x7f0520070dc0 2023-06-22T17:20:18.871 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.184+0000 7f0525243700 1 --2- 172.21.15.39:0/959588581 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0520070dc0 0x7f05201110e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f051c0094c0 tx=0x7f051c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:18.872 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.184+0000 7f0515ffb700 1 -- 172.21.15.39:0/959588581 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f051c01a760 con 0x7f0520070dc0 2023-06-22T17:20:18.872 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.184+0000 7f0515ffb700 1 -- 172.21.15.39:0/959588581 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f051c01ada0 con 0x7f0520070dc0 2023-06-22T17:20:18.872 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.184+0000 7f05274a7700 1 -- 172.21.15.39:0/959588581 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0520111820 con 0x7f0520070dc0 2023-06-22T17:20:18.872 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.184+0000 7f0515ffb700 1 -- 172.21.15.39:0/959588581 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f051c019cc0 con 0x7f0520070dc0 2023-06-22T17:20:18.872 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.184+0000 7f05274a7700 1 -- 172.21.15.39:0/959588581 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0520111c30 con 0x7f0520070dc0 2023-06-22T17:20:18.872 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.185+0000 7f0515ffb700 1 -- 172.21.15.39:0/959588581 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 8) v1 ==== 43219+0+0 (secure 0 0 0) 0x7f051c0233a0 con 0x7f0520070dc0 2023-06-22T17:20:18.872 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.185+0000 7f05274a7700 1 -- 172.21.15.39:0/959588581 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0520106590 con 0x7f0520070dc0 2023-06-22T17:20:18.873 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.185+0000 7f0515ffb700 1 --2- 172.21.15.39:0/959588581 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f050c036560 0x7f050c038a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:18.873 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.185+0000 7f0515ffb700 1 -- 172.21.15.39:0/959588581 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f051c04a100 con 0x7f0520070dc0 2023-06-22T17:20:18.873 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.186+0000 7f0524a42700 1 --2- 172.21.15.39:0/959588581 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f050c036560 0x7f050c038a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:18.873 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.186+0000 7f0524a42700 1 --2- 172.21.15.39:0/959588581 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f050c036560 0x7f050c038a10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0510009990 tx=0x7f0510006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:18.873 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.190+0000 7f0515ffb700 1 -- 172.21.15.39:0/959588581 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f051c0228f0 con 0x7f0520070dc0 2023-06-22T17:20:18.873 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.390+0000 7f05274a7700 1 -- 172.21.15.39:0/959588581 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f0520114690 con 0x7f0520070dc0 2023-06-22T17:20:18.873 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.398+0000 7f0515ffb700 1 -- 172.21.15.39:0/959588581 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f051c049a60 con 0x7f0520070dc0 2023-06-22T17:20:18.874 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.400+0000 7f05274a7700 1 -- 172.21.15.39:0/959588581 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f050c036560 msgr2=0x7f050c038a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:18.874 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.400+0000 7f05274a7700 1 --2- 172.21.15.39:0/959588581 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f050c036560 0x7f050c038a10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0510009990 tx=0x7f0510006e40 comp rx=0 tx=0).stop 2023-06-22T17:20:18.874 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.400+0000 7f05274a7700 1 -- 172.21.15.39:0/959588581 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0520070dc0 msgr2=0x7f05201110e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:18.874 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.400+0000 7f05274a7700 1 --2- 172.21.15.39:0/959588581 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0520070dc0 0x7f05201110e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f051c0094c0 tx=0x7f051c009250 comp rx=0 tx=0).stop 2023-06-22T17:20:18.874 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.400+0000 7f05274a7700 1 -- 172.21.15.39:0/959588581 shutdown_connections 2023-06-22T17:20:18.874 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.400+0000 7f05274a7700 1 --2- 172.21.15.39:0/959588581 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f050c036560 0x7f050c038a10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:18.874 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.400+0000 7f05274a7700 1 --2- 172.21.15.39:0/959588581 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0520070dc0 0x7f05201110e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:18.875 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.400+0000 7f05274a7700 1 -- 172.21.15.39:0/959588581 >> 172.21.15.39:0/959588581 conn(0x7f05200fca60 msgr2=0x7f05200fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:18.875 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.401+0000 7f05274a7700 1 -- 172.21.15.39:0/959588581 shutdown_connections 2023-06-22T17:20:18.875 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:18.401+0000 7f05274a7700 1 -- 172.21.15.39:0/959588581 wait complete. 2023-06-22T17:20:18.875 INFO:teuthology.orchestra.run.smithi039.stdout:Enabling the dashboard module... 2023-06-22T17:20:19.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:19 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/959588581' entity='client.admin' 2023-06-22T17:20:19.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:19 smithi039 ceph-mon[84805]: from='mgr.14118 172.21.15.39:0/1180832241' entity='mgr.x' 2023-06-22T17:20:20.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:20 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/3535937215' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-06-22T17:20:20.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:20 smithi039 ceph-mon[84805]: from='mgr.14118 172.21.15.39:0/1180832241' entity='mgr.x' 2023-06-22T17:20:20.683 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:20 smithi039 conmon[85217]: ignoring --setuser ceph since I am not root 2023-06-22T17:20:20.683 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:20 smithi039 conmon[85217]: ignoring --setgroup ceph since I am not root 2023-06-22T17:20:20.806 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.332+0000 7f4fd0070700 1 Processor -- start 2023-06-22T17:20:20.806 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.332+0000 7f4fd0070700 1 -- start start 2023-06-22T17:20:20.806 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.332+0000 7f4fd0070700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4fc8103a40 0x7f4fc8103e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:20.807 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.332+0000 7f4fd0070700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc8104390 con 0x7f4fc8103a40 2023-06-22T17:20:20.807 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.332+0000 7f4fcde0c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4fc8103a40 0x7f4fc8103e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:20.807 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.332+0000 7f4fcde0c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4fc8103a40 0x7f4fc8103e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35316/0 (socket says 172.21.15.39:35316) 2023-06-22T17:20:20.807 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.332+0000 7f4fcde0c700 1 -- 172.21.15.39:0/850295668 learned_addr learned my addr 172.21.15.39:0/850295668 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:20.807 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.333+0000 7f4fcde0c700 1 -- 172.21.15.39:0/850295668 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc81044d0 con 0x7f4fc8103a40 2023-06-22T17:20:20.807 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.333+0000 7f4fcde0c700 1 --2- 172.21.15.39:0/850295668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4fc8103a40 0x7f4fc8103e50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4fc40098e0 tx=0x7f4fc4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52bcb62a7bc5c340 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:20.807 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.333+0000 7f4fcce0a700 1 -- 172.21.15.39:0/850295668 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4fc4009e80 con 0x7f4fc8103a40 2023-06-22T17:20:20.808 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.333+0000 7f4fcce0a700 1 -- 172.21.15.39:0/850295668 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f4fc400bd40 con 0x7f4fc8103a40 2023-06-22T17:20:20.808 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.333+0000 7f4fcce0a700 1 -- 172.21.15.39:0/850295668 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4fc4011d10 con 0x7f4fc8103a40 2023-06-22T17:20:20.808 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.333+0000 7f4fd0070700 1 -- 172.21.15.39:0/850295668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4fc8103a40 msgr2=0x7f4fc8103e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:20.808 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.333+0000 7f4fd0070700 1 --2- 172.21.15.39:0/850295668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4fc8103a40 0x7f4fc8103e50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4fc40098e0 tx=0x7f4fc4009250 comp rx=0 tx=0).stop 2023-06-22T17:20:20.808 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.333+0000 7f4fd0070700 1 -- 172.21.15.39:0/850295668 shutdown_connections 2023-06-22T17:20:20.808 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.333+0000 7f4fd0070700 1 --2- 172.21.15.39:0/850295668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4fc8103a40 0x7f4fc8103e50 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:20.808 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.333+0000 7f4fd0070700 1 -- 172.21.15.39:0/850295668 >> 172.21.15.39:0/850295668 conn(0x7f4fc80fc1e0 msgr2=0x7f4fc80fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:20.808 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.334+0000 7f4fd0070700 1 -- 172.21.15.39:0/850295668 shutdown_connections 2023-06-22T17:20:20.809 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.334+0000 7f4fd0070700 1 -- 172.21.15.39:0/850295668 wait complete. 2023-06-22T17:20:20.809 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.334+0000 7f4fd0070700 1 Processor -- start 2023-06-22T17:20:20.809 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.334+0000 7f4fd0070700 1 -- start start 2023-06-22T17:20:20.809 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.334+0000 7f4fd0070700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4fc8103a40 0x7f4fc81110b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:20.809 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.334+0000 7f4fd0070700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc81115f0 con 0x7f4fc8103a40 2023-06-22T17:20:20.809 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.334+0000 7f4fcde0c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4fc8103a40 0x7f4fc81110b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:20.809 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.334+0000 7f4fcde0c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4fc8103a40 0x7f4fc81110b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35318/0 (socket says 172.21.15.39:35318) 2023-06-22T17:20:20.810 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.334+0000 7f4fcde0c700 1 -- 172.21.15.39:0/3535937215 learned_addr learned my addr 172.21.15.39:0/3535937215 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:20.810 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.335+0000 7f4fcde0c700 1 -- 172.21.15.39:0/3535937215 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc40094f0 con 0x7f4fc8103a40 2023-06-22T17:20:20.810 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.335+0000 7f4fcde0c700 1 --2- 172.21.15.39:0/3535937215 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4fc8103a40 0x7f4fc81110b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4fc40094c0 tx=0x7f4fc4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:20.810 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.335+0000 7f4fbe7fc700 1 -- 172.21.15.39:0/3535937215 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4fc401a760 con 0x7f4fc8103a40 2023-06-22T17:20:20.810 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.335+0000 7f4fd0070700 1 -- 172.21.15.39:0/3535937215 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fc81117f0 con 0x7f4fc8103a40 2023-06-22T17:20:20.810 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.335+0000 7f4fbe7fc700 1 -- 172.21.15.39:0/3535937215 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f4fc401ada0 con 0x7f4fc8103a40 2023-06-22T17:20:20.810 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.335+0000 7f4fbe7fc700 1 -- 172.21.15.39:0/3535937215 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4fc4019cf0 con 0x7f4fc8103a40 2023-06-22T17:20:20.810 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.335+0000 7f4fd0070700 1 -- 172.21.15.39:0/3535937215 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fc8111c00 con 0x7f4fc8103a40 2023-06-22T17:20:20.811 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.336+0000 7f4fbe7fc700 1 -- 172.21.15.39:0/3535937215 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 8) v1 ==== 43219+0+0 (secure 0 0 0) 0x7f4fc401a8c0 con 0x7f4fc8103a40 2023-06-22T17:20:20.811 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.336+0000 7f4fd0070700 1 -- 172.21.15.39:0/3535937215 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fc805f060 con 0x7f4fc8103a40 2023-06-22T17:20:20.811 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.336+0000 7f4fbe7fc700 1 --2- 172.21.15.39:0/3535937215 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f4fb4036560 0x7f4fb4038a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:20.811 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.336+0000 7f4fbe7fc700 1 -- 172.21.15.39:0/3535937215 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f4fc4049ef0 con 0x7f4fc8103a40 2023-06-22T17:20:20.811 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.336+0000 7f4fcd60b700 1 --2- 172.21.15.39:0/3535937215 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f4fb4036560 0x7f4fb4038a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:20.811 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.337+0000 7f4fcd60b700 1 --2- 172.21.15.39:0/3535937215 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f4fb4036560 0x7f4fb4038a10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4fb8009990 tx=0x7f4fb8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:20.811 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.341+0000 7f4fbe7fc700 1 -- 172.21.15.39:0/3535937215 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f4fc40195e0 con 0x7f4fc8103a40 2023-06-22T17:20:20.811 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:19.593+0000 7f4fd0070700 1 -- 172.21.15.39:0/3535937215 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f4fc8101040 con 0x7f4fc8103a40 2023-06-22T17:20:20.812 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:20.409+0000 7f4fbe7fc700 1 -- 172.21.15.39:0/3535937215 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mgrmap(e 9) v1 ==== 43232+0+0 (secure 0 0 0) 0x7f4fc40233a0 con 0x7f4fc8103a40 2023-06-22T17:20:20.812 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:20.410+0000 7f4fbe7fc700 1 -- 172.21.15.39:0/3535937215 <== mon.0 v2:172.21.15.39:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f4fc4041430 con 0x7f4fc8103a40 2023-06-22T17:20:20.812 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:20.412+0000 7f4fd0070700 1 -- 172.21.15.39:0/3535937215 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f4fb4036560 msgr2=0x7f4fb4038a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:20.812 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:20.412+0000 7f4fd0070700 1 --2- 172.21.15.39:0/3535937215 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f4fb4036560 0x7f4fb4038a10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4fb8009990 tx=0x7f4fb8006e40 comp rx=0 tx=0).stop 2023-06-22T17:20:20.812 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:20.412+0000 7f4fd0070700 1 -- 172.21.15.39:0/3535937215 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4fc8103a40 msgr2=0x7f4fc81110b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:20.812 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:20.412+0000 7f4fd0070700 1 --2- 172.21.15.39:0/3535937215 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4fc8103a40 0x7f4fc81110b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4fc40094c0 tx=0x7f4fc4009250 comp rx=0 tx=0).stop 2023-06-22T17:20:20.812 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:20.412+0000 7f4fd0070700 1 -- 172.21.15.39:0/3535937215 shutdown_connections 2023-06-22T17:20:20.813 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:20.412+0000 7f4fd0070700 1 --2- 172.21.15.39:0/3535937215 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f4fb4036560 0x7f4fb4038a10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:20.813 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:20.412+0000 7f4fd0070700 1 --2- 172.21.15.39:0/3535937215 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4fc8103a40 0x7f4fc81110b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:20.813 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:20.412+0000 7f4fd0070700 1 -- 172.21.15.39:0/3535937215 >> 172.21.15.39:0/3535937215 conn(0x7f4fc80fc1e0 msgr2=0x7f4fc80fcf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:20.813 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:20.412+0000 7f4fd0070700 1 -- 172.21.15.39:0/3535937215 shutdown_connections 2023-06-22T17:20:20.813 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:20.412+0000 7f4fd0070700 1 -- 172.21.15.39:0/3535937215 wait complete. 2023-06-22T17:20:21.678 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:21 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/3535937215' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-06-22T17:20:21.678 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:21 smithi039 ceph-mon[84805]: mgrmap e9: x(active, since 15s) 2023-06-22T17:20:21.932 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:21 smithi039 conmon[85217]: 2023-06-22T17:20:21.794+0000 7f0347eb41c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-06-22T17:20:22.059 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout { 2023-06-22T17:20:22.059 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-06-22T17:20:22.060 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "available": true, 2023-06-22T17:20:22.060 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-06-22T17:20:22.060 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-06-22T17:20:22.060 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout } 2023-06-22T17:20:22.060 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.313+0000 7fbf37869700 1 Processor -- start 2023-06-22T17:20:22.060 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.313+0000 7fbf37869700 1 -- start start 2023-06-22T17:20:22.060 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.314+0000 7fbf37869700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf30070dc0 0x7fbf30107410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:22.061 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.314+0000 7fbf37869700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf30102e00 con 0x7fbf30070dc0 2023-06-22T17:20:22.061 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.314+0000 7fbf35605700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf30070dc0 0x7fbf30107410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:22.061 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.314+0000 7fbf35605700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf30070dc0 0x7fbf30107410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35340/0 (socket says 172.21.15.39:35340) 2023-06-22T17:20:22.061 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.314+0000 7fbf35605700 1 -- 172.21.15.39:0/540549413 learned_addr learned my addr 172.21.15.39:0/540549413 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:22.061 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.314+0000 7fbf35605700 1 -- 172.21.15.39:0/540549413 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf30107950 con 0x7fbf30070dc0 2023-06-22T17:20:22.061 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.314+0000 7fbf35605700 1 --2- 172.21.15.39:0/540549413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf30070dc0 0x7fbf30107410 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c0098e0 tx=0x7fbf2c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc681eda3de68fc server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:22.061 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.314+0000 7fbf27fff700 1 -- 172.21.15.39:0/540549413 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbf2c009e80 con 0x7fbf30070dc0 2023-06-22T17:20:22.061 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.315+0000 7fbf27fff700 1 -- 172.21.15.39:0/540549413 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7fbf2c00bd40 con 0x7fbf30070dc0 2023-06-22T17:20:22.062 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.315+0000 7fbf27fff700 1 -- 172.21.15.39:0/540549413 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbf2c011d10 con 0x7fbf30070dc0 2023-06-22T17:20:22.062 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.315+0000 7fbf37869700 1 -- 172.21.15.39:0/540549413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf30070dc0 msgr2=0x7fbf30107410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:22.062 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.315+0000 7fbf37869700 1 --2- 172.21.15.39:0/540549413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf30070dc0 0x7fbf30107410 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c0098e0 tx=0x7fbf2c009250 comp rx=0 tx=0).stop 2023-06-22T17:20:22.062 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.315+0000 7fbf37869700 1 -- 172.21.15.39:0/540549413 shutdown_connections 2023-06-22T17:20:22.062 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.315+0000 7fbf37869700 1 --2- 172.21.15.39:0/540549413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf30070dc0 0x7fbf30107410 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:22.063 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.315+0000 7fbf37869700 1 -- 172.21.15.39:0/540549413 >> 172.21.15.39:0/540549413 conn(0x7fbf300fca60 msgr2=0x7fbf300fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:22.063 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.315+0000 7fbf37869700 1 -- 172.21.15.39:0/540549413 shutdown_connections 2023-06-22T17:20:22.063 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.315+0000 7fbf37869700 1 -- 172.21.15.39:0/540549413 wait complete. 2023-06-22T17:20:22.063 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.316+0000 7fbf37869700 1 Processor -- start 2023-06-22T17:20:22.063 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.316+0000 7fbf37869700 1 -- start start 2023-06-22T17:20:22.064 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.316+0000 7fbf37869700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf30070dc0 0x7fbf30113270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:22.064 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.316+0000 7fbf37869700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf301137b0 con 0x7fbf30070dc0 2023-06-22T17:20:22.064 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.316+0000 7fbf35605700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf30070dc0 0x7fbf30113270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:22.064 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.316+0000 7fbf35605700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf30070dc0 0x7fbf30113270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35356/0 (socket says 172.21.15.39:35356) 2023-06-22T17:20:22.065 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.316+0000 7fbf35605700 1 -- 172.21.15.39:0/3851998496 learned_addr learned my addr 172.21.15.39:0/3851998496 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:22.065 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.317+0000 7fbf35605700 1 -- 172.21.15.39:0/3851998496 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf2c0094f0 con 0x7fbf30070dc0 2023-06-22T17:20:22.065 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.317+0000 7fbf35605700 1 --2- 172.21.15.39:0/3851998496 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf30070dc0 0x7fbf30113270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c0094c0 tx=0x7fbf2c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:22.066 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.317+0000 7fbf25ffb700 1 -- 172.21.15.39:0/3851998496 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbf2c01a760 con 0x7fbf30070dc0 2023-06-22T17:20:22.066 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.317+0000 7fbf25ffb700 1 -- 172.21.15.39:0/3851998496 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7fbf2c01ada0 con 0x7fbf30070dc0 2023-06-22T17:20:22.066 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.317+0000 7fbf37869700 1 -- 172.21.15.39:0/3851998496 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf301139b0 con 0x7fbf30070dc0 2023-06-22T17:20:22.066 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.317+0000 7fbf25ffb700 1 -- 172.21.15.39:0/3851998496 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbf2c019cc0 con 0x7fbf30070dc0 2023-06-22T17:20:22.067 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.317+0000 7fbf37869700 1 -- 172.21.15.39:0/3851998496 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf30113dc0 con 0x7fbf30070dc0 2023-06-22T17:20:22.067 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.318+0000 7fbf25ffb700 1 -- 172.21.15.39:0/3851998496 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 9) v1 ==== 43232+0+0 (secure 0 0 0) 0x7fbf2c0233a0 con 0x7fbf30070dc0 2023-06-22T17:20:22.067 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.318+0000 7fbf25ffb700 1 --2- 172.21.15.39:0/3851998496 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7fbf1c0365c0 0x7fbf1c038a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:22.067 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.318+0000 7fbf25ffb700 1 -- 172.21.15.39:0/3851998496 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fbf2c04ae50 con 0x7fbf30070dc0 2023-06-22T17:20:22.068 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.318+0000 7fbf34e04700 1 -- 172.21.15.39:0/3851998496 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7fbf1c0365c0 msgr2=0x7fbf1c038a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/1350590065 2023-06-22T17:20:22.068 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.318+0000 7fbf37869700 1 -- 172.21.15.39:0/3851998496 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf30108790 con 0x7fbf30070dc0 2023-06-22T17:20:22.068 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.318+0000 7fbf34e04700 1 --2- 172.21.15.39:0/3851998496 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7fbf1c0365c0 0x7fbf1c038a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T17:20:22.069 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.324+0000 7fbf25ffb700 1 -- 172.21.15.39:0/3851998496 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fbf2c0228f0 con 0x7fbf30070dc0 2023-06-22T17:20:22.069 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.519+0000 7fbf34e04700 1 -- 172.21.15.39:0/3851998496 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7fbf1c0365c0 msgr2=0x7fbf1c038a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/1350590065 2023-06-22T17:20:22.069 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.519+0000 7fbf34e04700 1 --2- 172.21.15.39:0/3851998496 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7fbf1c0365c0 0x7fbf1c038a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T17:20:22.069 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.594+0000 7fbf37869700 1 -- 172.21.15.39:0/3851998496 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fbf30116820 con 0x7fbf30070dc0 2023-06-22T17:20:22.069 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.594+0000 7fbf25ffb700 1 -- 172.21.15.39:0/3851998496 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7fbf30116820 con 0x7fbf30070dc0 2023-06-22T17:20:22.069 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.597+0000 7fbf37869700 1 -- 172.21.15.39:0/3851998496 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7fbf1c0365c0 msgr2=0x7fbf1c038a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-06-22T17:20:22.069 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.597+0000 7fbf37869700 1 --2- 172.21.15.39:0/3851998496 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7fbf1c0365c0 0x7fbf1c038a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:22.070 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.597+0000 7fbf37869700 1 -- 172.21.15.39:0/3851998496 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf30070dc0 msgr2=0x7fbf30113270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:22.070 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.597+0000 7fbf37869700 1 --2- 172.21.15.39:0/3851998496 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf30070dc0 0x7fbf30113270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c0094c0 tx=0x7fbf2c009250 comp rx=0 tx=0).stop 2023-06-22T17:20:22.070 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.597+0000 7fbf37869700 1 -- 172.21.15.39:0/3851998496 shutdown_connections 2023-06-22T17:20:22.070 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.597+0000 7fbf37869700 1 --2- 172.21.15.39:0/3851998496 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7fbf1c0365c0 0x7fbf1c038a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:22.070 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.597+0000 7fbf37869700 1 --2- 172.21.15.39:0/3851998496 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf30070dc0 0x7fbf30113270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:22.070 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.597+0000 7fbf37869700 1 -- 172.21.15.39:0/3851998496 >> 172.21.15.39:0/3851998496 conn(0x7fbf300fca60 msgr2=0x7fbf300fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:22.070 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.597+0000 7fbf37869700 1 -- 172.21.15.39:0/3851998496 shutdown_connections 2023-06-22T17:20:22.071 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:21.597+0000 7fbf37869700 1 -- 172.21.15.39:0/3851998496 wait complete. 2023-06-22T17:20:22.071 INFO:teuthology.orchestra.run.smithi039.stdout:Waiting for the mgr to restart... 2023-06-22T17:20:22.071 INFO:teuthology.orchestra.run.smithi039.stdout:Waiting for mgr epoch 9... 2023-06-22T17:20:22.241 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:21 smithi039 conmon[85217]: 2023-06-22T17:20:21.965+0000 7f0347eb41c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-06-22T17:20:22.241 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:22 smithi039 conmon[85217]: 2023-06-22T17:20:22.060+0000 7f0347eb41c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-06-22T17:20:22.512 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:22 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/3851998496' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-06-22T17:20:22.513 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:22 smithi039 conmon[85217]: 2023-06-22T17:20:22.295+0000 7f0347eb41c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-06-22T17:20:22.763 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:22 smithi039 conmon[85217]: 2023-06-22T17:20:22.511+0000 7f0347eb41c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-06-22T17:20:22.763 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:22 smithi039 conmon[85217]: 2023-06-22T17:20:22.678+0000 7f0347eb41c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-06-22T17:20:22.763 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:22 smithi039 conmon[85217]: 2023-06-22T17:20:22.762+0000 7f0347eb41c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-06-22T17:20:23.682 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:23 smithi039 conmon[85217]: 2023-06-22T17:20:23.377+0000 7f0347eb41c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-06-22T17:20:24.182 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:23 smithi039 conmon[85217]: 2023-06-22T17:20:23.885+0000 7f0347eb41c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-06-22T17:20:24.182 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:23 smithi039 conmon[85217]: 2023-06-22T17:20:23.982+0000 7f0347eb41c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-06-22T17:20:24.182 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:24 smithi039 conmon[85217]: 2023-06-22T17:20:24.078+0000 7f0347eb41c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-06-22T17:20:24.973 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:24 smithi039 conmon[85217]: 2023-06-22T17:20:24.698+0000 7f0347eb41c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-06-22T17:20:24.974 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:24 smithi039 conmon[85217]: 2023-06-22T17:20:24.876+0000 7f0347eb41c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-06-22T17:20:25.432 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:24 smithi039 conmon[85217]: 2023-06-22T17:20:24.972+0000 7f0347eb41c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-06-22T17:20:25.432 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:25 smithi039 conmon[85217]: 2023-06-22T17:20:25.109+0000 7f0347eb41c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-06-22T17:20:25.432 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:25 smithi039 conmon[85217]: 2023-06-22T17:20:25.205+0000 7f0347eb41c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-06-22T17:20:26.432 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:26 smithi039 conmon[85217]: 2023-06-22T17:20:26.130+0000 7f0347eb41c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-06-22T17:20:26.432 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:26 smithi039 conmon[85217]: 2023-06-22T17:20:26.223+0000 7f0347eb41c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-06-22T17:20:26.432 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:26 smithi039 conmon[85217]: 2023-06-22T17:20:26.316+0000 7f0347eb41c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-06-22T17:20:26.932 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:26 smithi039 conmon[85217]: 2023-06-22T17:20:26.473+0000 7f0347eb41c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-06-22T17:20:26.932 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:26 smithi039 conmon[85217]: 2023-06-22T17:20:26.577+0000 7f0347eb41c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-06-22T17:20:27.268 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:26 smithi039 conmon[85217]: 2023-06-22T17:20:26.985+0000 7f0347eb41c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-06-22T17:20:27.269 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:27 smithi039 conmon[85217]: 2023-06-22T17:20:27.079+0000 7f0347eb41c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-06-22T17:20:27.269 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:27 smithi039 conmon[85217]: 2023-06-22T17:20:27.173+0000 7f0347eb41c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-06-22T17:20:27.682 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:20:27 smithi039 conmon[85217]: 2023-06-22T17:20:27.267+0000 7f0347eb41c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-06-22T17:20:27.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:27 smithi039 ceph-mon[84805]: Active manager daemon x restarted 2023-06-22T17:20:27.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:27 smithi039 ceph-mon[84805]: Activating manager daemon x 2023-06-22T17:20:27.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:27 smithi039 ceph-mon[84805]: osdmap e3: 0 total, 0 up, 0 in 2023-06-22T17:20:27.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:27 smithi039 ceph-mon[84805]: mgrmap e10: x(active, starting, since 0.00546205s) 2023-06-22T17:20:27.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:27 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-22T17:20:27.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:27 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-06-22T17:20:27.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:27 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-06-22T17:20:27.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:27 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-06-22T17:20:27.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:27 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-06-22T17:20:27.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:27 smithi039 ceph-mon[84805]: Manager daemon x is now available 2023-06-22T17:20:27.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:27 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:20:27.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:27 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:20:28.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:28 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-06-22T17:20:28.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:28 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-06-22T17:20:28.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:28 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:28.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:28 smithi039 ceph-mon[84805]: mgrmap e11: x(active, since 1.00777s) 2023-06-22T17:20:28.801 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout { 2023-06-22T17:20:28.806 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-06-22T17:20:28.806 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "initialized": true 2023-06-22T17:20:28.807 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout } 2023-06-22T17:20:28.807 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.491+0000 7f22c28f8700 1 Processor -- start 2023-06-22T17:20:28.807 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.491+0000 7f22c28f8700 1 -- start start 2023-06-22T17:20:28.807 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.491+0000 7f22c28f8700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22bc070d80 0x7f22bc105050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:28.807 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.491+0000 7f22c28f8700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22bc105590 con 0x7f22bc070d80 2023-06-22T17:20:28.807 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.491+0000 7f22bbfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22bc070d80 0x7f22bc105050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:28.808 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.491+0000 7f22bbfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22bc070d80 0x7f22bc105050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35362/0 (socket says 172.21.15.39:35362) 2023-06-22T17:20:28.808 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.491+0000 7f22bbfff700 1 -- 172.21.15.39:0/52451024 learned_addr learned my addr 172.21.15.39:0/52451024 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:28.808 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.491+0000 7f22bbfff700 1 -- 172.21.15.39:0/52451024 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22bc1056d0 con 0x7f22bc070d80 2023-06-22T17:20:28.808 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.491+0000 7f22bbfff700 1 --2- 172.21.15.39:0/52451024 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22bc070d80 0x7f22bc105050 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f22ac0098e0 tx=0x7f22ac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8c11802b71d4492 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:28.809 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.492+0000 7f22baffd700 1 -- 172.21.15.39:0/52451024 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f22ac009e80 con 0x7f22bc070d80 2023-06-22T17:20:28.809 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.492+0000 7f22baffd700 1 -- 172.21.15.39:0/52451024 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f22ac00bd40 con 0x7f22bc070d80 2023-06-22T17:20:28.809 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.492+0000 7f22baffd700 1 -- 172.21.15.39:0/52451024 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f22ac011d10 con 0x7f22bc070d80 2023-06-22T17:20:28.809 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.492+0000 7f22c28f8700 1 -- 172.21.15.39:0/52451024 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22bc070d80 msgr2=0x7f22bc105050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:28.809 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.492+0000 7f22c28f8700 1 --2- 172.21.15.39:0/52451024 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22bc070d80 0x7f22bc105050 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f22ac0098e0 tx=0x7f22ac009250 comp rx=0 tx=0).stop 2023-06-22T17:20:28.809 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.492+0000 7f22c28f8700 1 -- 172.21.15.39:0/52451024 shutdown_connections 2023-06-22T17:20:28.809 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.492+0000 7f22c28f8700 1 --2- 172.21.15.39:0/52451024 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22bc070d80 0x7f22bc105050 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:28.809 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.492+0000 7f22c28f8700 1 -- 172.21.15.39:0/52451024 >> 172.21.15.39:0/52451024 conn(0x7f22bc0fca60 msgr2=0x7f22bc0fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:28.810 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.492+0000 7f22c28f8700 1 -- 172.21.15.39:0/52451024 shutdown_connections 2023-06-22T17:20:28.810 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.492+0000 7f22c28f8700 1 -- 172.21.15.39:0/52451024 wait complete. 2023-06-22T17:20:28.810 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.493+0000 7f22c28f8700 1 Processor -- start 2023-06-22T17:20:28.810 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.493+0000 7f22c28f8700 1 -- start start 2023-06-22T17:20:28.810 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.493+0000 7f22c28f8700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22bc070d80 0x7f22bc1028f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:28.810 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.493+0000 7f22c28f8700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22bc102e30 con 0x7f22bc070d80 2023-06-22T17:20:28.810 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.493+0000 7f22bbfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22bc070d80 0x7f22bc1028f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:28.810 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.493+0000 7f22bbfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22bc070d80 0x7f22bc1028f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35374/0 (socket says 172.21.15.39:35374) 2023-06-22T17:20:28.811 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.493+0000 7f22bbfff700 1 -- 172.21.15.39:0/1239787151 learned_addr learned my addr 172.21.15.39:0/1239787151 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:28.811 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.494+0000 7f22bbfff700 1 -- 172.21.15.39:0/1239787151 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22ac0094f0 con 0x7f22bc070d80 2023-06-22T17:20:28.811 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.494+0000 7f22bbfff700 1 --2- 172.21.15.39:0/1239787151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22bc070d80 0x7f22bc1028f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f22ac0094c0 tx=0x7f22ac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:28.811 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.494+0000 7f22b8ff9700 1 -- 172.21.15.39:0/1239787151 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f22ac01a760 con 0x7f22bc070d80 2023-06-22T17:20:28.811 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.494+0000 7f22b8ff9700 1 -- 172.21.15.39:0/1239787151 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f22ac01ada0 con 0x7f22bc070d80 2023-06-22T17:20:28.811 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.494+0000 7f22c28f8700 1 -- 172.21.15.39:0/1239787151 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22bc100f50 con 0x7f22bc070d80 2023-06-22T17:20:28.811 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.494+0000 7f22b8ff9700 1 -- 172.21.15.39:0/1239787151 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f22ac019cc0 con 0x7f22bc070d80 2023-06-22T17:20:28.812 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.494+0000 7f22c28f8700 1 -- 172.21.15.39:0/1239787151 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22bc1013c0 con 0x7f22bc070d80 2023-06-22T17:20:28.812 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.495+0000 7f22b8ff9700 1 -- 172.21.15.39:0/1239787151 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 9) v1 ==== 43232+0+0 (secure 0 0 0) 0x7f22ac0233a0 con 0x7f22bc070d80 2023-06-22T17:20:28.812 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.495+0000 7f22b8ff9700 1 --2- 172.21.15.39:0/1239787151 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f22a4036610 0x7f22a4038ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:28.812 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.495+0000 7f22c28f8700 1 -- 172.21.15.39:0/1239787151 --> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f22bc05f060 con 0x7f22a4036610 2023-06-22T17:20:28.812 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.495+0000 7f22b8ff9700 1 -- 172.21.15.39:0/1239787151 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f22ac049f00 con 0x7f22bc070d80 2023-06-22T17:20:28.812 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.495+0000 7f22bb7fe700 1 -- 172.21.15.39:0/1239787151 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f22a4036610 msgr2=0x7f22a4038ac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/1350590065 2023-06-22T17:20:28.813 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.495+0000 7f22bb7fe700 1 --2- 172.21.15.39:0/1239787151 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f22a4036610 0x7f22a4038ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T17:20:28.813 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.696+0000 7f22bb7fe700 1 -- 172.21.15.39:0/1239787151 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f22a4036610 msgr2=0x7f22a4038ac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/1350590065 2023-06-22T17:20:28.813 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:22.696+0000 7f22bb7fe700 1 --2- 172.21.15.39:0/1239787151 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f22a4036610 0x7f22a4038ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T17:20:28.813 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:23.097+0000 7f22bb7fe700 1 -- 172.21.15.39:0/1239787151 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f22a4036610 msgr2=0x7f22a4038ac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/1350590065 2023-06-22T17:20:28.813 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:23.097+0000 7f22bb7fe700 1 --2- 172.21.15.39:0/1239787151 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f22a4036610 0x7f22a4038ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T17:20:28.813 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:23.898+0000 7f22bb7fe700 1 -- 172.21.15.39:0/1239787151 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f22a4036610 msgr2=0x7f22a4038ac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/1350590065 2023-06-22T17:20:28.813 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:23.898+0000 7f22bb7fe700 1 --2- 172.21.15.39:0/1239787151 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f22a4036610 0x7f22a4038ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T17:20:28.814 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:25.499+0000 7f22bb7fe700 1 -- 172.21.15.39:0/1239787151 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f22a4036610 msgr2=0x7f22a4038ac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/1350590065 2023-06-22T17:20:28.814 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:25.499+0000 7f22bb7fe700 1 --2- 172.21.15.39:0/1239787151 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f22a4036610 0x7f22a4038ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-06-22T17:20:28.814 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:27.285+0000 7f22b8ff9700 1 -- 172.21.15.39:0/1239787151 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mgrmap(e 10) v1 ==== 42999+0+0 (secure 0 0 0) 0x7f22ac02c350 con 0x7f22bc070d80 2023-06-22T17:20:28.814 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:27.285+0000 7f22b8ff9700 1 -- 172.21.15.39:0/1239787151 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f22a4036610 msgr2=0x7f22a4038ac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-06-22T17:20:28.814 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:27.285+0000 7f22b8ff9700 1 --2- 172.21.15.39:0/1239787151 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f22a4036610 0x7f22a4038ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:28.814 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:28.287+0000 7f22b8ff9700 1 -- 172.21.15.39:0/1239787151 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mgrmap(e 11) v1 ==== 43126+0+0 (secure 0 0 0) 0x7f22ac02bc00 con 0x7f22bc070d80 2023-06-22T17:20:28.814 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:28.287+0000 7f22b8ff9700 1 --2- 172.21.15.39:0/1239787151 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f22a4039ec0 0x7f22a403c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:28.814 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:28.287+0000 7f22b8ff9700 1 -- 172.21.15.39:0/1239787151 --> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f22bc05f060 con 0x7f22a4039ec0 2023-06-22T17:20:28.815 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:28.287+0000 7f22bb7fe700 1 --2- 172.21.15.39:0/1239787151 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f22a4039ec0 0x7f22a403c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:28.815 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:28.288+0000 7f22bb7fe700 1 --2- 172.21.15.39:0/1239787151 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f22a4039ec0 0x7f22a403c2a0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f22b00039d0 tx=0x7f22b0009350 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:28.815 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:28.288+0000 7f22b8ff9700 1 -- 172.21.15.39:0/1239787151 <== mgr.14150 v2:172.21.15.39:6800/886078282 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f22bc05f060 con 0x7f22a4039ec0 2023-06-22T17:20:28.815 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:28.294+0000 7f22c28f8700 1 -- 172.21.15.39:0/1239787151 --> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f22bc106570 con 0x7f22a4039ec0 2023-06-22T17:20:28.815 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:28.294+0000 7f22b8ff9700 1 -- 172.21.15.39:0/1239787151 <== mgr.14150 v2:172.21.15.39:6800/886078282 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f22bc106570 con 0x7f22a4039ec0 2023-06-22T17:20:28.815 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:28.295+0000 7f22c28f8700 1 -- 172.21.15.39:0/1239787151 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f22a4039ec0 msgr2=0x7f22a403c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:28.815 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:28.295+0000 7f22c28f8700 1 --2- 172.21.15.39:0/1239787151 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f22a4039ec0 0x7f22a403c2a0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f22b00039d0 tx=0x7f22b0009350 comp rx=0 tx=0).stop 2023-06-22T17:20:28.816 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:28.295+0000 7f22c28f8700 1 -- 172.21.15.39:0/1239787151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22bc070d80 msgr2=0x7f22bc1028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:28.816 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:28.295+0000 7f22c28f8700 1 --2- 172.21.15.39:0/1239787151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22bc070d80 0x7f22bc1028f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f22ac0094c0 tx=0x7f22ac009250 comp rx=0 tx=0).stop 2023-06-22T17:20:28.816 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:28.295+0000 7f22c28f8700 1 -- 172.21.15.39:0/1239787151 shutdown_connections 2023-06-22T17:20:28.816 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:28.295+0000 7f22c28f8700 1 --2- 172.21.15.39:0/1239787151 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f22a4039ec0 0x7f22a403c2a0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:28.816 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:28.295+0000 7f22c28f8700 1 --2- 172.21.15.39:0/1239787151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22bc070d80 0x7f22bc1028f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:28.816 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:28.295+0000 7f22c28f8700 1 --2- 172.21.15.39:0/1239787151 >> [v2:172.21.15.39:6800/1350590065,v1:172.21.15.39:6801/1350590065] conn(0x7f22a4036610 0x7f22a4038ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:28.816 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:28.295+0000 7f22c28f8700 1 -- 172.21.15.39:0/1239787151 >> 172.21.15.39:0/1239787151 conn(0x7f22bc0fca60 msgr2=0x7f22bc0fd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:28.817 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:28.295+0000 7f22c28f8700 1 -- 172.21.15.39:0/1239787151 shutdown_connections 2023-06-22T17:20:28.817 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:28.295+0000 7f22c28f8700 1 -- 172.21.15.39:0/1239787151 wait complete. 2023-06-22T17:20:28.817 INFO:teuthology.orchestra.run.smithi039.stdout:mgr epoch 9 is available 2023-06-22T17:20:28.817 INFO:teuthology.orchestra.run.smithi039.stdout:Generating a dashboard self-signed certificate... 2023-06-22T17:20:29.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:29 smithi039 ceph-mon[84805]: [22/Jun/2023:17:20:27] ENGINE Bus STARTING 2023-06-22T17:20:29.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:29 smithi039 ceph-mon[84805]: [22/Jun/2023:17:20:27] ENGINE Serving on https://172.21.15.39:7150 2023-06-22T17:20:29.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:29 smithi039 ceph-mon[84805]: [22/Jun/2023:17:20:27] ENGINE Bus STARTED 2023-06-22T17:20:29.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:29 smithi039 ceph-mon[84805]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-06-22T17:20:29.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:29 smithi039 ceph-mon[84805]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-06-22T17:20:29.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:29 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:30.299 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-06-22T17:20:30.300 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.517+0000 7ff12173f700 1 Processor -- start 2023-06-22T17:20:30.300 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.517+0000 7ff12173f700 1 -- start start 2023-06-22T17:20:30.300 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.517+0000 7ff12173f700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff11c070dc0 0x7ff11c105010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:30.300 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.517+0000 7ff12173f700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff11c105550 con 0x7ff11c070dc0 2023-06-22T17:20:30.300 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.517+0000 7ff11affd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff11c070dc0 0x7ff11c105010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:30.300 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.517+0000 7ff11affd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff11c070dc0 0x7ff11c105010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36488/0 (socket says 172.21.15.39:36488) 2023-06-22T17:20:30.301 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.517+0000 7ff11affd700 1 -- 172.21.15.39:0/3296934209 learned_addr learned my addr 172.21.15.39:0/3296934209 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:30.301 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.518+0000 7ff11affd700 1 -- 172.21.15.39:0/3296934209 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff11c105690 con 0x7ff11c070dc0 2023-06-22T17:20:30.301 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.518+0000 7ff11affd700 1 --2- 172.21.15.39:0/3296934209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff11c070dc0 0x7ff11c105010 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff1140098e0 tx=0x7ff114009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=821be440e1df4d25 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:30.301 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.518+0000 7ff119ffb700 1 -- 172.21.15.39:0/3296934209 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff114009e80 con 0x7ff11c070dc0 2023-06-22T17:20:30.301 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.518+0000 7ff119ffb700 1 -- 172.21.15.39:0/3296934209 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7ff11400bd40 con 0x7ff11c070dc0 2023-06-22T17:20:30.301 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.518+0000 7ff119ffb700 1 -- 172.21.15.39:0/3296934209 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff114011d10 con 0x7ff11c070dc0 2023-06-22T17:20:30.301 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.519+0000 7ff12173f700 1 -- 172.21.15.39:0/3296934209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff11c070dc0 msgr2=0x7ff11c105010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:30.301 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.519+0000 7ff12173f700 1 --2- 172.21.15.39:0/3296934209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff11c070dc0 0x7ff11c105010 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff1140098e0 tx=0x7ff114009250 comp rx=0 tx=0).stop 2023-06-22T17:20:30.302 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.519+0000 7ff12173f700 1 -- 172.21.15.39:0/3296934209 shutdown_connections 2023-06-22T17:20:30.302 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.519+0000 7ff12173f700 1 --2- 172.21.15.39:0/3296934209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff11c070dc0 0x7ff11c105010 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:30.302 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.519+0000 7ff12173f700 1 -- 172.21.15.39:0/3296934209 >> 172.21.15.39:0/3296934209 conn(0x7ff11c0fca60 msgr2=0x7ff11c0fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:30.302 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.519+0000 7ff12173f700 1 -- 172.21.15.39:0/3296934209 shutdown_connections 2023-06-22T17:20:30.302 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.519+0000 7ff12173f700 1 -- 172.21.15.39:0/3296934209 wait complete. 2023-06-22T17:20:30.302 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.519+0000 7ff12173f700 1 Processor -- start 2023-06-22T17:20:30.302 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.519+0000 7ff12173f700 1 -- start start 2023-06-22T17:20:30.303 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.520+0000 7ff12173f700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff11c070dc0 0x7ff11c1110e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:30.303 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.520+0000 7ff12173f700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff11c111620 con 0x7ff11c070dc0 2023-06-22T17:20:30.303 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.520+0000 7ff11affd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff11c070dc0 0x7ff11c1110e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:30.303 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.520+0000 7ff11affd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff11c070dc0 0x7ff11c1110e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36504/0 (socket says 172.21.15.39:36504) 2023-06-22T17:20:30.303 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.520+0000 7ff11affd700 1 -- 172.21.15.39:0/1961065288 learned_addr learned my addr 172.21.15.39:0/1961065288 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:30.303 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.520+0000 7ff11affd700 1 -- 172.21.15.39:0/1961065288 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1140094f0 con 0x7ff11c070dc0 2023-06-22T17:20:30.303 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.520+0000 7ff11affd700 1 --2- 172.21.15.39:0/1961065288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff11c070dc0 0x7ff11c1110e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff11400bd10 tx=0x7ff114009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:30.304 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.520+0000 7ff1137fe700 1 -- 172.21.15.39:0/1961065288 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff11401a760 con 0x7ff11c070dc0 2023-06-22T17:20:30.304 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.520+0000 7ff1137fe700 1 -- 172.21.15.39:0/1961065288 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7ff11401ada0 con 0x7ff11c070dc0 2023-06-22T17:20:30.304 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.520+0000 7ff12173f700 1 -- 172.21.15.39:0/1961065288 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff11c111820 con 0x7ff11c070dc0 2023-06-22T17:20:30.304 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.520+0000 7ff1137fe700 1 -- 172.21.15.39:0/1961065288 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff114024c80 con 0x7ff11c070dc0 2023-06-22T17:20:30.304 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.520+0000 7ff12173f700 1 -- 172.21.15.39:0/1961065288 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff11c111c30 con 0x7ff11c070dc0 2023-06-22T17:20:30.304 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.521+0000 7ff1137fe700 1 -- 172.21.15.39:0/1961065288 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 11) v1 ==== 43126+0+0 (secure 0 0 0) 0x7ff11402d3f0 con 0x7ff11c070dc0 2023-06-22T17:20:30.304 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.521+0000 7ff12173f700 1 -- 172.21.15.39:0/1961065288 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff11c05f060 con 0x7ff11c070dc0 2023-06-22T17:20:30.305 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.521+0000 7ff1137fe700 1 --2- 172.21.15.39:0/1961065288 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7ff1080364a0 0x7ff108038950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:30.305 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.521+0000 7ff1137fe700 1 -- 172.21.15.39:0/1961065288 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7ff11404a100 con 0x7ff11c070dc0 2023-06-22T17:20:30.305 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.522+0000 7ff11a7fc700 1 --2- 172.21.15.39:0/1961065288 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7ff1080364a0 0x7ff108038950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:30.305 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.522+0000 7ff11a7fc700 1 --2- 172.21.15.39:0/1961065288 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7ff1080364a0 0x7ff108038950 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff104009990 tx=0x7ff104006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:30.305 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.526+0000 7ff1137fe700 1 -- 172.21.15.39:0/1961065288 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7ff1140189a0 con 0x7ff11c070dc0 2023-06-22T17:20:30.305 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.728+0000 7ff12173f700 1 -- 172.21.15.39:0/1961065288 --> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7ff11c0fdf60 con 0x7ff1080364a0 2023-06-22T17:20:30.305 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.822+0000 7ff1137fe700 1 -- 172.21.15.39:0/1961065288 <== mgr.14150 v2:172.21.15.39:6800/886078282 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7ff11c0fdf60 con 0x7ff1080364a0 2023-06-22T17:20:30.306 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.824+0000 7ff12173f700 1 -- 172.21.15.39:0/1961065288 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7ff1080364a0 msgr2=0x7ff108038950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:30.306 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.824+0000 7ff12173f700 1 --2- 172.21.15.39:0/1961065288 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7ff1080364a0 0x7ff108038950 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff104009990 tx=0x7ff104006e40 comp rx=0 tx=0).stop 2023-06-22T17:20:30.306 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.824+0000 7ff12173f700 1 -- 172.21.15.39:0/1961065288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff11c070dc0 msgr2=0x7ff11c1110e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:30.306 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.824+0000 7ff12173f700 1 --2- 172.21.15.39:0/1961065288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff11c070dc0 0x7ff11c1110e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff11400bd10 tx=0x7ff114009250 comp rx=0 tx=0).stop 2023-06-22T17:20:30.306 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.824+0000 7ff12173f700 1 -- 172.21.15.39:0/1961065288 shutdown_connections 2023-06-22T17:20:30.306 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.824+0000 7ff12173f700 1 --2- 172.21.15.39:0/1961065288 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7ff1080364a0 0x7ff108038950 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:30.306 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.824+0000 7ff12173f700 1 --2- 172.21.15.39:0/1961065288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff11c070dc0 0x7ff11c1110e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:30.307 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.824+0000 7ff12173f700 1 -- 172.21.15.39:0/1961065288 >> 172.21.15.39:0/1961065288 conn(0x7ff11c0fca60 msgr2=0x7ff11c0fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:30.307 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.824+0000 7ff12173f700 1 -- 172.21.15.39:0/1961065288 shutdown_connections 2023-06-22T17:20:30.307 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:29.824+0000 7ff12173f700 1 -- 172.21.15.39:0/1961065288 wait complete. 2023-06-22T17:20:30.307 INFO:teuthology.orchestra.run.smithi039.stdout:Creating initial admin user... 2023-06-22T17:20:31.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:30 smithi039 ceph-mon[84805]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:20:31.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:30 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:31.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:30 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:31.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:30 smithi039 ceph-mon[84805]: mgrmap e12: x(active, since 2s) 2023-06-22T17:20:31.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:30 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:31.839 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$YWfh9Ui0FevaroPEuF9Wt.Xa2TlENara5akFBHMNkC2IBXCqNxFM2", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1687454431, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-06-22T17:20:31.839 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.832+0000 7ff2059a6700 1 Processor -- start 2023-06-22T17:20:31.839 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.832+0000 7ff2059a6700 1 -- start start 2023-06-22T17:20:31.840 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.832+0000 7ff2059a6700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff200103ae0 0x7ff200103ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:31.840 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.832+0000 7ff2059a6700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff200104430 con 0x7ff200103ae0 2023-06-22T17:20:31.840 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.832+0000 7ff1feffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff200103ae0 0x7ff200103ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:31.840 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.832+0000 7ff1feffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff200103ae0 0x7ff200103ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36512/0 (socket says 172.21.15.39:36512) 2023-06-22T17:20:31.840 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.832+0000 7ff1feffd700 1 -- 172.21.15.39:0/3315947906 learned_addr learned my addr 172.21.15.39:0/3315947906 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:31.840 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.833+0000 7ff1feffd700 1 -- 172.21.15.39:0/3315947906 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff200104570 con 0x7ff200103ae0 2023-06-22T17:20:31.840 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.833+0000 7ff1feffd700 1 --2- 172.21.15.39:0/3315947906 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff200103ae0 0x7ff200103ef0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff1f00098e0 tx=0x7ff1f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59e26fdfdab5598c server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:31.841 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.833+0000 7ff1fdffb700 1 -- 172.21.15.39:0/3315947906 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff1f0009e80 con 0x7ff200103ae0 2023-06-22T17:20:31.841 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.833+0000 7ff1fdffb700 1 -- 172.21.15.39:0/3315947906 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7ff1f000bd40 con 0x7ff200103ae0 2023-06-22T17:20:31.841 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.833+0000 7ff1fdffb700 1 -- 172.21.15.39:0/3315947906 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff1f0011d10 con 0x7ff200103ae0 2023-06-22T17:20:31.841 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.833+0000 7ff2059a6700 1 -- 172.21.15.39:0/3315947906 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff200103ae0 msgr2=0x7ff200103ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:31.841 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.833+0000 7ff2059a6700 1 --2- 172.21.15.39:0/3315947906 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff200103ae0 0x7ff200103ef0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff1f00098e0 tx=0x7ff1f0009250 comp rx=0 tx=0).stop 2023-06-22T17:20:31.841 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.833+0000 7ff2059a6700 1 -- 172.21.15.39:0/3315947906 shutdown_connections 2023-06-22T17:20:31.841 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.833+0000 7ff2059a6700 1 --2- 172.21.15.39:0/3315947906 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff200103ae0 0x7ff200103ef0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:31.842 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.833+0000 7ff2059a6700 1 -- 172.21.15.39:0/3315947906 >> 172.21.15.39:0/3315947906 conn(0x7ff2000fc1e0 msgr2=0x7ff2000fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:31.842 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.833+0000 7ff2059a6700 1 -- 172.21.15.39:0/3315947906 shutdown_connections 2023-06-22T17:20:31.842 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.834+0000 7ff2059a6700 1 -- 172.21.15.39:0/3315947906 wait complete. 2023-06-22T17:20:31.842 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.834+0000 7ff2059a6700 1 Processor -- start 2023-06-22T17:20:31.842 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.834+0000 7ff2059a6700 1 -- start start 2023-06-22T17:20:31.842 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.834+0000 7ff2059a6700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff200103ae0 0x7ff2001110a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:31.842 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.834+0000 7ff2059a6700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2001115e0 con 0x7ff200103ae0 2023-06-22T17:20:31.842 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.834+0000 7ff1feffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff200103ae0 0x7ff2001110a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:31.843 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.834+0000 7ff1feffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff200103ae0 0x7ff2001110a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36516/0 (socket says 172.21.15.39:36516) 2023-06-22T17:20:31.843 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.834+0000 7ff1feffd700 1 -- 172.21.15.39:0/3334863273 learned_addr learned my addr 172.21.15.39:0/3334863273 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:31.843 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.835+0000 7ff1feffd700 1 -- 172.21.15.39:0/3334863273 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f00094f0 con 0x7ff200103ae0 2023-06-22T17:20:31.843 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.835+0000 7ff1feffd700 1 --2- 172.21.15.39:0/3334863273 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff200103ae0 0x7ff2001110a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0006b00 tx=0x7ff1f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:31.843 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.835+0000 7ff1e77fe700 1 -- 172.21.15.39:0/3334863273 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff1f001a710 con 0x7ff200103ae0 2023-06-22T17:20:31.843 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.835+0000 7ff1e77fe700 1 -- 172.21.15.39:0/3334863273 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7ff1f001ad50 con 0x7ff200103ae0 2023-06-22T17:20:31.843 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.835+0000 7ff2059a6700 1 -- 172.21.15.39:0/3334863273 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2001117e0 con 0x7ff200103ae0 2023-06-22T17:20:31.844 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.835+0000 7ff1e77fe700 1 -- 172.21.15.39:0/3334863273 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff1f0024c50 con 0x7ff200103ae0 2023-06-22T17:20:31.844 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.835+0000 7ff2059a6700 1 -- 172.21.15.39:0/3334863273 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff200111bf0 con 0x7ff200103ae0 2023-06-22T17:20:31.844 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.836+0000 7ff1e77fe700 1 -- 172.21.15.39:0/3334863273 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 12) v1 ==== 43232+0+0 (secure 0 0 0) 0x7ff1f00183d0 con 0x7ff200103ae0 2023-06-22T17:20:31.844 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.836+0000 7ff1e77fe700 1 --2- 172.21.15.39:0/3334863273 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7ff1e8036610 0x7ff1e8038ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:31.844 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.836+0000 7ff1e77fe700 1 -- 172.21.15.39:0/3334863273 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7ff1f004a520 con 0x7ff200103ae0 2023-06-22T17:20:31.844 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.836+0000 7ff2059a6700 1 -- 172.21.15.39:0/3334863273 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff20005f060 con 0x7ff200103ae0 2023-06-22T17:20:31.844 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.836+0000 7ff1fe7fc700 1 --2- 172.21.15.39:0/3334863273 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7ff1e8036610 0x7ff1e8038ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:31.844 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.837+0000 7ff1fe7fc700 1 --2- 172.21.15.39:0/3334863273 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7ff1e8036610 0x7ff1e8038ac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4009990 tx=0x7ff1f4006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:31.845 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:30.841+0000 7ff1e77fe700 1 -- 172.21.15.39:0/3334863273 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7ff1f002c8f0 con 0x7ff200103ae0 2023-06-22T17:20:31.845 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:31.071+0000 7ff2059a6700 1 -- 172.21.15.39:0/3334863273 --> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7ff20005ee80 con 0x7ff1e8036610 2023-06-22T17:20:31.845 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:31.320+0000 7ff1e77fe700 1 -- 172.21.15.39:0/3334863273 <== mgr.14150 v2:172.21.15.39:6800/886078282 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7ff20005ee80 con 0x7ff1e8036610 2023-06-22T17:20:31.845 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:31.322+0000 7ff2059a6700 1 -- 172.21.15.39:0/3334863273 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7ff1e8036610 msgr2=0x7ff1e8038ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:31.845 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:31.322+0000 7ff2059a6700 1 --2- 172.21.15.39:0/3334863273 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7ff1e8036610 0x7ff1e8038ac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4009990 tx=0x7ff1f4006e40 comp rx=0 tx=0).stop 2023-06-22T17:20:31.845 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:31.322+0000 7ff2059a6700 1 -- 172.21.15.39:0/3334863273 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff200103ae0 msgr2=0x7ff2001110a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:31.846 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:31.322+0000 7ff2059a6700 1 --2- 172.21.15.39:0/3334863273 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff200103ae0 0x7ff2001110a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0006b00 tx=0x7ff1f0009250 comp rx=0 tx=0).stop 2023-06-22T17:20:31.846 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:31.322+0000 7ff2059a6700 1 -- 172.21.15.39:0/3334863273 shutdown_connections 2023-06-22T17:20:31.846 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:31.322+0000 7ff2059a6700 1 --2- 172.21.15.39:0/3334863273 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7ff1e8036610 0x7ff1e8038ac0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:31.846 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:31.322+0000 7ff2059a6700 1 --2- 172.21.15.39:0/3334863273 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff200103ae0 0x7ff2001110a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:31.846 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:31.322+0000 7ff2059a6700 1 -- 172.21.15.39:0/3334863273 >> 172.21.15.39:0/3334863273 conn(0x7ff2000fc1e0 msgr2=0x7ff2000fcf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:31.846 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:31.322+0000 7ff2059a6700 1 -- 172.21.15.39:0/3334863273 shutdown_connections 2023-06-22T17:20:31.846 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:31.322+0000 7ff2059a6700 1 -- 172.21.15.39:0/3334863273 wait complete. 2023-06-22T17:20:31.847 INFO:teuthology.orchestra.run.smithi039.stdout:Fetching dashboard port number... 2023-06-22T17:20:32.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:32 smithi039 ceph-mon[84805]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:20:32.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:32 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:33.178 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 8443 2023-06-22T17:20:33.179 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.587+0000 7f0f0ac5e700 1 Processor -- start 2023-06-22T17:20:33.179 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.587+0000 7f0f0ac5e700 1 -- start start 2023-06-22T17:20:33.179 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.588+0000 7f0f0ac5e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f04102400 0x7f0f04102810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:33.179 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.588+0000 7f0f0ac5e700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f04102d50 con 0x7f0f04102400 2023-06-22T17:20:33.179 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.588+0000 7f0f089fa700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f04102400 0x7f0f04102810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:33.179 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.588+0000 7f0f089fa700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f04102400 0x7f0f04102810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36524/0 (socket says 172.21.15.39:36524) 2023-06-22T17:20:33.180 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.588+0000 7f0f089fa700 1 -- 172.21.15.39:0/1168185130 learned_addr learned my addr 172.21.15.39:0/1168185130 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:33.180 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.588+0000 7f0f089fa700 1 -- 172.21.15.39:0/1168185130 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f04102e90 con 0x7f0f04102400 2023-06-22T17:20:33.180 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.588+0000 7f0f089fa700 1 --2- 172.21.15.39:0/1168185130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f04102400 0x7f0f04102810 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0efc009e50 tx=0x7f0efc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99a8d6aed0ac04bf server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:33.180 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.588+0000 7f0f037fe700 1 -- 172.21.15.39:0/1168185130 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0efc00e070 con 0x7f0f04102400 2023-06-22T17:20:33.180 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.588+0000 7f0f037fe700 1 -- 172.21.15.39:0/1168185130 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f0efc012430 con 0x7f0f04102400 2023-06-22T17:20:33.180 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.588+0000 7f0f037fe700 1 -- 172.21.15.39:0/1168185130 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0efc007400 con 0x7f0f04102400 2023-06-22T17:20:33.180 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.589+0000 7f0f0ac5e700 1 -- 172.21.15.39:0/1168185130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f04102400 msgr2=0x7f0f04102810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:33.180 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.589+0000 7f0f0ac5e700 1 --2- 172.21.15.39:0/1168185130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f04102400 0x7f0f04102810 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0efc009e50 tx=0x7f0efc009840 comp rx=0 tx=0).stop 2023-06-22T17:20:33.181 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.589+0000 7f0f0ac5e700 1 -- 172.21.15.39:0/1168185130 shutdown_connections 2023-06-22T17:20:33.181 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.589+0000 7f0f0ac5e700 1 --2- 172.21.15.39:0/1168185130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f04102400 0x7f0f04102810 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:33.181 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.589+0000 7f0f0ac5e700 1 -- 172.21.15.39:0/1168185130 >> 172.21.15.39:0/1168185130 conn(0x7f0f040fdb80 msgr2=0x7f0f040fff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:33.181 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.589+0000 7f0f0ac5e700 1 -- 172.21.15.39:0/1168185130 shutdown_connections 2023-06-22T17:20:33.181 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.589+0000 7f0f0ac5e700 1 -- 172.21.15.39:0/1168185130 wait complete. 2023-06-22T17:20:33.181 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.590+0000 7f0f0ac5e700 1 Processor -- start 2023-06-22T17:20:33.181 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.590+0000 7f0f0ac5e700 1 -- start start 2023-06-22T17:20:33.182 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.590+0000 7f0f0ac5e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f04102400 0x7f0f04111080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:33.182 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.590+0000 7f0f0ac5e700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f041115c0 con 0x7f0f04102400 2023-06-22T17:20:33.182 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.590+0000 7f0f089fa700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f04102400 0x7f0f04111080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:33.182 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.590+0000 7f0f089fa700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f04102400 0x7f0f04111080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36534/0 (socket says 172.21.15.39:36534) 2023-06-22T17:20:33.182 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.590+0000 7f0f089fa700 1 -- 172.21.15.39:0/532274276 learned_addr learned my addr 172.21.15.39:0/532274276 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:33.182 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.590+0000 7f0f089fa700 1 -- 172.21.15.39:0/532274276 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0efc009ae0 con 0x7f0f04102400 2023-06-22T17:20:33.182 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.590+0000 7f0f089fa700 1 --2- 172.21.15.39:0/532274276 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f04102400 0x7f0f04111080 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0efc0127d0 tx=0x7f0efc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:33.182 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.591+0000 7f0f017fa700 1 -- 172.21.15.39:0/532274276 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0efc00e070 con 0x7f0f04102400 2023-06-22T17:20:33.183 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.591+0000 7f0f017fa700 1 -- 172.21.15.39:0/532274276 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f0efc00bd50 con 0x7f0f04102400 2023-06-22T17:20:33.183 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.591+0000 7f0f0ac5e700 1 -- 172.21.15.39:0/532274276 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f041117c0 con 0x7f0f04102400 2023-06-22T17:20:33.183 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.591+0000 7f0f017fa700 1 -- 172.21.15.39:0/532274276 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0efc0258f0 con 0x7f0f04102400 2023-06-22T17:20:33.183 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.591+0000 7f0f0ac5e700 1 -- 172.21.15.39:0/532274276 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f04111bd0 con 0x7f0f04102400 2023-06-22T17:20:33.183 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.591+0000 7f0f017fa700 1 -- 172.21.15.39:0/532274276 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 12) v1 ==== 43232+0+0 (secure 0 0 0) 0x7f0efc019050 con 0x7f0f04102400 2023-06-22T17:20:33.183 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.592+0000 7f0f017fa700 1 --2- 172.21.15.39:0/532274276 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f0eec036610 0x7f0eec038ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:33.183 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.592+0000 7f0f017fa700 1 -- 172.21.15.39:0/532274276 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0efc04a010 con 0x7f0f04102400 2023-06-22T17:20:33.183 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.592+0000 7f0f03fff700 1 --2- 172.21.15.39:0/532274276 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f0eec036610 0x7f0eec038ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:33.184 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.592+0000 7f0f0ac5e700 1 -- 172.21.15.39:0/532274276 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f0405f060 con 0x7f0f04102400 2023-06-22T17:20:33.184 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.592+0000 7f0f03fff700 1 --2- 172.21.15.39:0/532274276 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f0eec036610 0x7f0eec038ac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8009990 tx=0x7f0ef8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:33.184 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.597+0000 7f0f017fa700 1 -- 172.21.15.39:0/532274276 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f0efc017070 con 0x7f0f04102400 2023-06-22T17:20:33.184 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.793+0000 7f0f0ac5e700 1 -- 172.21.15.39:0/532274276 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f0f041147b0 con 0x7f0f04102400 2023-06-22T17:20:33.184 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.793+0000 7f0f017fa700 1 -- 172.21.15.39:0/532274276 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f0efc020030 con 0x7f0f04102400 2023-06-22T17:20:33.184 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.795+0000 7f0f0ac5e700 1 -- 172.21.15.39:0/532274276 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f0eec036610 msgr2=0x7f0eec038ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:33.184 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.795+0000 7f0f0ac5e700 1 --2- 172.21.15.39:0/532274276 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f0eec036610 0x7f0eec038ac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8009990 tx=0x7f0ef8006e40 comp rx=0 tx=0).stop 2023-06-22T17:20:33.185 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.795+0000 7f0f0ac5e700 1 -- 172.21.15.39:0/532274276 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f04102400 msgr2=0x7f0f04111080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:33.185 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.795+0000 7f0f0ac5e700 1 --2- 172.21.15.39:0/532274276 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f04102400 0x7f0f04111080 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0efc0127d0 tx=0x7f0efc009840 comp rx=0 tx=0).stop 2023-06-22T17:20:33.185 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.795+0000 7f0f0ac5e700 1 -- 172.21.15.39:0/532274276 shutdown_connections 2023-06-22T17:20:33.185 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.795+0000 7f0f0ac5e700 1 --2- 172.21.15.39:0/532274276 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f0eec036610 0x7f0eec038ac0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:33.185 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.795+0000 7f0f0ac5e700 1 --2- 172.21.15.39:0/532274276 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f04102400 0x7f0f04111080 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:33.185 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.795+0000 7f0f0ac5e700 1 -- 172.21.15.39:0/532274276 >> 172.21.15.39:0/532274276 conn(0x7f0f040fdb80 msgr2=0x7f0f040fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:33.185 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.796+0000 7f0f0ac5e700 1 -- 172.21.15.39:0/532274276 shutdown_connections 2023-06-22T17:20:33.185 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:32.796+0000 7f0f0ac5e700 1 -- 172.21.15.39:0/532274276 wait complete. 2023-06-22T17:20:33.187 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-06-22T17:20:33.187 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout disabled 2023-06-22T17:20:33.195 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-06-22T17:20:33.195 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout inactive 2023-06-22T17:20:33.195 INFO:teuthology.orchestra.run.smithi039.stdout:firewalld.service is not enabled 2023-06-22T17:20:33.195 INFO:teuthology.orchestra.run.smithi039.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-06-22T17:20:33.197 INFO:teuthology.orchestra.run.smithi039.stdout:Ceph Dashboard is now available at: 2023-06-22T17:20:33.198 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:20:33.198 INFO:teuthology.orchestra.run.smithi039.stdout: URL: https://smithi039.front.sepia.ceph.com:8443/ 2023-06-22T17:20:33.198 INFO:teuthology.orchestra.run.smithi039.stdout: User: admin 2023-06-22T17:20:33.198 INFO:teuthology.orchestra.run.smithi039.stdout: Password: jfeybqj82o 2023-06-22T17:20:33.198 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:20:33.199 INFO:teuthology.orchestra.run.smithi039.stdout:Saving cluster configuration to /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/config directory 2023-06-22T17:20:33.199 INFO:teuthology.orchestra.run.smithi039.stdout:Enabling autotune for osd_memory_target 2023-06-22T17:20:33.398 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:33 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/532274276' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-06-22T17:20:34.242 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.646+0000 7ff4b9cc1700 1 Processor -- start 2023-06-22T17:20:34.242 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.646+0000 7ff4b9cc1700 1 -- start start 2023-06-22T17:20:34.243 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.646+0000 7ff4b9cc1700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4b4070de0 0x7ff4b4104ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:34.243 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.646+0000 7ff4b9cc1700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b4105530 con 0x7ff4b4070de0 2023-06-22T17:20:34.243 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.647+0000 7ff4b37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4b4070de0 0x7ff4b4104ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:34.243 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.647+0000 7ff4b37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4b4070de0 0x7ff4b4104ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36538/0 (socket says 172.21.15.39:36538) 2023-06-22T17:20:34.244 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.647+0000 7ff4b37fe700 1 -- 172.21.15.39:0/3512440558 learned_addr learned my addr 172.21.15.39:0/3512440558 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:34.244 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.647+0000 7ff4b37fe700 1 -- 172.21.15.39:0/3512440558 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b4105670 con 0x7ff4b4070de0 2023-06-22T17:20:34.244 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.647+0000 7ff4b37fe700 1 --2- 172.21.15.39:0/3512440558 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4b4070de0 0x7ff4b4104ff0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac0098e0 tx=0x7ff4ac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=318dc853e07f9a69 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:34.245 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.647+0000 7ff4b27fc700 1 -- 172.21.15.39:0/3512440558 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff4ac009e80 con 0x7ff4b4070de0 2023-06-22T17:20:34.245 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.647+0000 7ff4b27fc700 1 -- 172.21.15.39:0/3512440558 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7ff4ac00bd40 con 0x7ff4b4070de0 2023-06-22T17:20:34.245 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.647+0000 7ff4b27fc700 1 -- 172.21.15.39:0/3512440558 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff4ac011d10 con 0x7ff4b4070de0 2023-06-22T17:20:34.245 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.648+0000 7ff4b9cc1700 1 -- 172.21.15.39:0/3512440558 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4b4070de0 msgr2=0x7ff4b4104ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:34.246 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.648+0000 7ff4b9cc1700 1 --2- 172.21.15.39:0/3512440558 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4b4070de0 0x7ff4b4104ff0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac0098e0 tx=0x7ff4ac009250 comp rx=0 tx=0).stop 2023-06-22T17:20:34.246 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.648+0000 7ff4b9cc1700 1 -- 172.21.15.39:0/3512440558 shutdown_connections 2023-06-22T17:20:34.246 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.648+0000 7ff4b9cc1700 1 --2- 172.21.15.39:0/3512440558 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4b4070de0 0x7ff4b4104ff0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:34.246 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.648+0000 7ff4b9cc1700 1 -- 172.21.15.39:0/3512440558 >> 172.21.15.39:0/3512440558 conn(0x7ff4b40fca60 msgr2=0x7ff4b40fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:34.246 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.648+0000 7ff4b9cc1700 1 -- 172.21.15.39:0/3512440558 shutdown_connections 2023-06-22T17:20:34.246 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.648+0000 7ff4b9cc1700 1 -- 172.21.15.39:0/3512440558 wait complete. 2023-06-22T17:20:34.247 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.648+0000 7ff4b9cc1700 1 Processor -- start 2023-06-22T17:20:34.247 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.649+0000 7ff4b9cc1700 1 -- start start 2023-06-22T17:20:34.247 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.649+0000 7ff4b9cc1700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4b4070de0 0x7ff4b41110c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:34.247 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.649+0000 7ff4b9cc1700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b4111600 con 0x7ff4b4070de0 2023-06-22T17:20:34.247 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.649+0000 7ff4b37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4b4070de0 0x7ff4b41110c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:34.247 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.649+0000 7ff4b37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4b4070de0 0x7ff4b41110c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36542/0 (socket says 172.21.15.39:36542) 2023-06-22T17:20:34.247 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.649+0000 7ff4b37fe700 1 -- 172.21.15.39:0/1136124473 learned_addr learned my addr 172.21.15.39:0/1136124473 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:34.247 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.649+0000 7ff4b37fe700 1 -- 172.21.15.39:0/1136124473 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4ac0094f0 con 0x7ff4b4070de0 2023-06-22T17:20:34.248 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.649+0000 7ff4b37fe700 1 --2- 172.21.15.39:0/1136124473 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4b4070de0 0x7ff4b41110c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac0094c0 tx=0x7ff4ac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:34.248 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.649+0000 7ff4abfff700 1 -- 172.21.15.39:0/1136124473 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff4ac01a760 con 0x7ff4b4070de0 2023-06-22T17:20:34.248 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.649+0000 7ff4abfff700 1 -- 172.21.15.39:0/1136124473 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7ff4ac01ada0 con 0x7ff4b4070de0 2023-06-22T17:20:34.248 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.649+0000 7ff4b9cc1700 1 -- 172.21.15.39:0/1136124473 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4b4111800 con 0x7ff4b4070de0 2023-06-22T17:20:34.248 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.649+0000 7ff4abfff700 1 -- 172.21.15.39:0/1136124473 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff4ac019cc0 con 0x7ff4b4070de0 2023-06-22T17:20:34.248 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.650+0000 7ff4b9cc1700 1 -- 172.21.15.39:0/1136124473 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4b4111c10 con 0x7ff4b4070de0 2023-06-22T17:20:34.248 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.650+0000 7ff4abfff700 1 -- 172.21.15.39:0/1136124473 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) v1 ==== 43275+0+0 (secure 0 0 0) 0x7ff4ac0233a0 con 0x7ff4b4070de0 2023-06-22T17:20:34.249 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.650+0000 7ff4b9cc1700 1 -- 172.21.15.39:0/1136124473 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4b405f060 con 0x7ff4b4070de0 2023-06-22T17:20:34.249 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.650+0000 7ff4abfff700 1 --2- 172.21.15.39:0/1136124473 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7ff4a0036660 0x7ff4a0038b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:34.249 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.650+0000 7ff4abfff700 1 -- 172.21.15.39:0/1136124473 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7ff4ac04a220 con 0x7ff4b4070de0 2023-06-22T17:20:34.249 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.651+0000 7ff4b2ffd700 1 --2- 172.21.15.39:0/1136124473 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7ff4a0036660 0x7ff4a0038b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:34.249 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.651+0000 7ff4b2ffd700 1 --2- 172.21.15.39:0/1136124473 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7ff4a0036660 0x7ff4a0038b10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff49c009990 tx=0x7ff49c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:34.249 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.656+0000 7ff4abfff700 1 -- 172.21.15.39:0/1136124473 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7ff4ac02bcd0 con 0x7ff4b4070de0 2023-06-22T17:20:34.249 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.846+0000 7ff4b9cc1700 1 -- 172.21.15.39:0/1136124473 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7ff4b4106570 con 0x7ff4b4070de0 2023-06-22T17:20:34.249 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.846+0000 7ff4abfff700 1 -- 172.21.15.39:0/1136124473 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7ff4ac049ab0 con 0x7ff4b4070de0 2023-06-22T17:20:34.250 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.848+0000 7ff4b9cc1700 1 -- 172.21.15.39:0/1136124473 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7ff4a0036660 msgr2=0x7ff4a0038b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:34.250 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.848+0000 7ff4b9cc1700 1 --2- 172.21.15.39:0/1136124473 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7ff4a0036660 0x7ff4a0038b10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff49c009990 tx=0x7ff49c006e40 comp rx=0 tx=0).stop 2023-06-22T17:20:34.250 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.848+0000 7ff4b9cc1700 1 -- 172.21.15.39:0/1136124473 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4b4070de0 msgr2=0x7ff4b41110c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:34.250 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.848+0000 7ff4b9cc1700 1 --2- 172.21.15.39:0/1136124473 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4b4070de0 0x7ff4b41110c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac0094c0 tx=0x7ff4ac009250 comp rx=0 tx=0).stop 2023-06-22T17:20:34.250 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.848+0000 7ff4b9cc1700 1 -- 172.21.15.39:0/1136124473 shutdown_connections 2023-06-22T17:20:34.250 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.848+0000 7ff4b9cc1700 1 --2- 172.21.15.39:0/1136124473 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7ff4a0036660 0x7ff4a0038b10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:34.250 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.848+0000 7ff4b9cc1700 1 --2- 172.21.15.39:0/1136124473 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4b4070de0 0x7ff4b41110c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:34.251 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.848+0000 7ff4b9cc1700 1 -- 172.21.15.39:0/1136124473 >> 172.21.15.39:0/1136124473 conn(0x7ff4b40fca60 msgr2=0x7ff4b40fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:34.251 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.848+0000 7ff4b9cc1700 1 -- 172.21.15.39:0/1136124473 shutdown_connections 2023-06-22T17:20:34.251 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:33.848+0000 7ff4b9cc1700 1 -- 172.21.15.39:0/1136124473 wait complete. 2023-06-22T17:20:34.490 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:34 smithi039 ceph-mon[84805]: mgrmap e13: x(active, since 6s) 2023-06-22T17:20:35.379 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.686+0000 7f8594eca700 1 Processor -- start 2023-06-22T17:20:35.380 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.686+0000 7f8594eca700 1 -- start start 2023-06-22T17:20:35.380 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.686+0000 7f8594eca700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8590078c80 0x7f8590077190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:35.380 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.686+0000 7f8594eca700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8590079090 con 0x7f8590078c80 2023-06-22T17:20:35.380 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.686+0000 7f858e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8590078c80 0x7f8590077190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:35.381 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.686+0000 7f858e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8590078c80 0x7f8590077190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36556/0 (socket says 172.21.15.39:36556) 2023-06-22T17:20:35.381 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.686+0000 7f858e59c700 1 -- 172.21.15.39:0/741913532 learned_addr learned my addr 172.21.15.39:0/741913532 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:35.381 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.687+0000 7f858e59c700 1 -- 172.21.15.39:0/741913532 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85900776d0 con 0x7f8590078c80 2023-06-22T17:20:35.381 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.687+0000 7f858e59c700 1 --2- 172.21.15.39:0/741913532 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8590078c80 0x7f8590077190 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f85880098e0 tx=0x7f8588009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=289c101319570b72 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:35.381 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.687+0000 7f858d59a700 1 -- 172.21.15.39:0/741913532 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8588009e80 con 0x7f8590078c80 2023-06-22T17:20:35.381 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.687+0000 7f858d59a700 1 -- 172.21.15.39:0/741913532 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f858800bd40 con 0x7f8590078c80 2023-06-22T17:20:35.381 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.687+0000 7f858d59a700 1 -- 172.21.15.39:0/741913532 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8588011d10 con 0x7f8590078c80 2023-06-22T17:20:35.382 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.688+0000 7f8594eca700 1 -- 172.21.15.39:0/741913532 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8590078c80 msgr2=0x7f8590077190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:35.382 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.688+0000 7f8594eca700 1 --2- 172.21.15.39:0/741913532 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8590078c80 0x7f8590077190 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f85880098e0 tx=0x7f8588009250 comp rx=0 tx=0).stop 2023-06-22T17:20:35.382 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.688+0000 7f8594eca700 1 -- 172.21.15.39:0/741913532 shutdown_connections 2023-06-22T17:20:35.382 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.688+0000 7f8594eca700 1 --2- 172.21.15.39:0/741913532 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8590078c80 0x7f8590077190 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:35.382 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.688+0000 7f8594eca700 1 -- 172.21.15.39:0/741913532 >> 172.21.15.39:0/741913532 conn(0x7f85900fdbc0 msgr2=0x7f85900fffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:35.382 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.688+0000 7f8594eca700 1 -- 172.21.15.39:0/741913532 shutdown_connections 2023-06-22T17:20:35.383 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.688+0000 7f8594eca700 1 -- 172.21.15.39:0/741913532 wait complete. 2023-06-22T17:20:35.383 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.688+0000 7f8594eca700 1 Processor -- start 2023-06-22T17:20:35.383 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.688+0000 7f8594eca700 1 -- start start 2023-06-22T17:20:35.383 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.688+0000 7f8594eca700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8590078c80 0x7f85901132b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:35.383 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.688+0000 7f8594eca700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85901137f0 con 0x7f8590078c80 2023-06-22T17:20:35.383 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.689+0000 7f858e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8590078c80 0x7f85901132b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:35.383 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.689+0000 7f858e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8590078c80 0x7f85901132b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36566/0 (socket says 172.21.15.39:36566) 2023-06-22T17:20:35.383 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.689+0000 7f858e59c700 1 -- 172.21.15.39:0/259628903 learned_addr learned my addr 172.21.15.39:0/259628903 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:35.384 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.689+0000 7f858e59c700 1 -- 172.21.15.39:0/259628903 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85880094f0 con 0x7f8590078c80 2023-06-22T17:20:35.384 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.689+0000 7f858e59c700 1 --2- 172.21.15.39:0/259628903 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8590078c80 0x7f85901132b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f85880094c0 tx=0x7f8588009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:35.384 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.689+0000 7f8586ffd700 1 -- 172.21.15.39:0/259628903 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f858801a760 con 0x7f8590078c80 2023-06-22T17:20:35.384 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.689+0000 7f8586ffd700 1 -- 172.21.15.39:0/259628903 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f858801ada0 con 0x7f8590078c80 2023-06-22T17:20:35.384 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.689+0000 7f8586ffd700 1 -- 172.21.15.39:0/259628903 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8588019cc0 con 0x7f8590078c80 2023-06-22T17:20:35.384 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.689+0000 7f8594eca700 1 -- 172.21.15.39:0/259628903 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85901139f0 con 0x7f8590078c80 2023-06-22T17:20:35.384 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.689+0000 7f8594eca700 1 -- 172.21.15.39:0/259628903 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8590113e00 con 0x7f8590078c80 2023-06-22T17:20:35.385 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.690+0000 7f8586ffd700 1 -- 172.21.15.39:0/259628903 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f858801a8c0 con 0x7f8590078c80 2023-06-22T17:20:35.385 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.690+0000 7f8594eca700 1 -- 172.21.15.39:0/259628903 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f859005f060 con 0x7f8590078c80 2023-06-22T17:20:35.385 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.690+0000 7f8586ffd700 1 --2- 172.21.15.39:0/259628903 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f857c036660 0x7f857c038b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:35.385 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.690+0000 7f8586ffd700 1 -- 172.21.15.39:0/259628903 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f858804a110 con 0x7f8590078c80 2023-06-22T17:20:35.385 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.690+0000 7f858dd9b700 1 --2- 172.21.15.39:0/259628903 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f857c036660 0x7f857c038b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:35.385 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.691+0000 7f858dd9b700 1 --2- 172.21.15.39:0/259628903 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f857c036660 0x7f857c038b10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8578009990 tx=0x7f8578006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:35.385 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:34.695+0000 7f8586ffd700 1 -- 172.21.15.39:0/259628903 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f85880193c0 con 0x7f8590078c80 2023-06-22T17:20:35.385 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:35.000+0000 7f8594eca700 1 -- 172.21.15.39:0/259628903 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f8590108810 con 0x7f8590078c80 2023-06-22T17:20:35.386 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:35.008+0000 7f8586ffd700 1 -- 172.21.15.39:0/259628903 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7f85880499a0 con 0x7f8590078c80 2023-06-22T17:20:35.386 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:35.009+0000 7f8594eca700 1 -- 172.21.15.39:0/259628903 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f857c036660 msgr2=0x7f857c038b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:35.386 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:35.009+0000 7f8594eca700 1 --2- 172.21.15.39:0/259628903 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f857c036660 0x7f857c038b10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8578009990 tx=0x7f8578006e40 comp rx=0 tx=0).stop 2023-06-22T17:20:35.386 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:35.009+0000 7f8594eca700 1 -- 172.21.15.39:0/259628903 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8590078c80 msgr2=0x7f85901132b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:35.387 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:35.009+0000 7f8594eca700 1 --2- 172.21.15.39:0/259628903 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8590078c80 0x7f85901132b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f85880094c0 tx=0x7f8588009250 comp rx=0 tx=0).stop 2023-06-22T17:20:35.387 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:35.010+0000 7f8594eca700 1 -- 172.21.15.39:0/259628903 shutdown_connections 2023-06-22T17:20:35.387 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:35.010+0000 7f8594eca700 1 --2- 172.21.15.39:0/259628903 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f857c036660 0x7f857c038b10 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:35.387 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:35.010+0000 7f8594eca700 1 --2- 172.21.15.39:0/259628903 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8590078c80 0x7f85901132b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:35.388 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:35.010+0000 7f8594eca700 1 -- 172.21.15.39:0/259628903 >> 172.21.15.39:0/259628903 conn(0x7f85900fdbc0 msgr2=0x7f85900fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:35.388 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:35.010+0000 7f8594eca700 1 -- 172.21.15.39:0/259628903 shutdown_connections 2023-06-22T17:20:35.388 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2023-06-22T17:20:35.010+0000 7f8594eca700 1 -- 172.21.15.39:0/259628903 wait complete. 2023-06-22T17:20:35.388 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-06-22T17:20:35.388 INFO:teuthology.orchestra.run.smithi039.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-06-22T17:20:35.389 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:20:35.389 INFO:teuthology.orchestra.run.smithi039.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-06-22T17:20:35.389 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:20:35.389 INFO:teuthology.orchestra.run.smithi039.stdout:Or, if you are only running a single cluster on this host: 2023-06-22T17:20:35.389 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:20:35.389 INFO:teuthology.orchestra.run.smithi039.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2023-06-22T17:20:35.389 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:20:35.389 INFO:teuthology.orchestra.run.smithi039.stdout:Please consider enabling telemetry to help improve Ceph: 2023-06-22T17:20:35.390 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:20:35.390 INFO:teuthology.orchestra.run.smithi039.stdout: ceph telemetry on 2023-06-22T17:20:35.390 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:20:35.390 INFO:teuthology.orchestra.run.smithi039.stdout:For more information see: 2023-06-22T17:20:35.390 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:20:35.390 INFO:teuthology.orchestra.run.smithi039.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2023-06-22T17:20:35.390 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:20:35.391 INFO:teuthology.orchestra.run.smithi039.stdout:Bootstrap complete. 2023-06-22T17:20:35.415 INFO:tasks.cephadm:Fetching config... 2023-06-22T17:20:35.416 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-22T17:20:35.417 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-06-22T17:20:35.432 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-06-22T17:20:35.433 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-22T17:20:35.433 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-06-22T17:20:35.488 INFO:tasks.cephadm:Fetching mon keyring... 2023-06-22T17:20:35.489 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-22T17:20:35.489 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.a/keyring of=/dev/stdout 2023-06-22T17:20:35.556 INFO:tasks.cephadm:Fetching pub ssh key... 2023-06-22T17:20:35.556 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-22T17:20:35.556 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-06-22T17:20:35.611 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-06-22T17:20:35.612 DEBUG:teuthology.orchestra.run.smithi039:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5tQ9nDpBOSN+wtaWGNclkt7PkTanv9ZMInGcJeS0T8zZaeDxvKTlyvYR1jrtDQQxaF24a+cbnEn1sXhp7nu5lZ8smN/uwt2MCOO+yn0B2tCvMwBxmGjHnSxd5AyXscuIUituiObHRDirchwuGlZ3+HWqvlcAI2lXOrA/8GxXi0s9qqtkx7RBG8mNpWEZrUhibb9rl9vrQp9D4QGBzyiumUSJzGsd3VXXI8MavE6woeGMw1fH+seIDI8qYbqClG6y3+BOoLlZFspZqAE2TXT+fFXSzw571QCOQ5nWHNxZAiv5DwqqeYrbnIY++Veh4NLhl6vpVfgqqKlUIiEue7JVl+8zA9KHJQqLskT9moip5RO+3elKBpDa0hjcd7cCeeCjYY2oGbKf38OL07bZIMlBNmNVSy7qWPti1bbNOFjczpBPXMamXH2jskzSip/9ummOolAB8duTm6L7T7k+/1tYOz6IXQGgPaP5Xu41FBPIhQfwJFugBen5zhLK8rRzpJ1U= ceph-d88002e2-1120-11ee-9b2b-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-06-22T17:20:35.696 INFO:teuthology.orchestra.run.smithi039.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5tQ9nDpBOSN+wtaWGNclkt7PkTanv9ZMInGcJeS0T8zZaeDxvKTlyvYR1jrtDQQxaF24a+cbnEn1sXhp7nu5lZ8smN/uwt2MCOO+yn0B2tCvMwBxmGjHnSxd5AyXscuIUituiObHRDirchwuGlZ3+HWqvlcAI2lXOrA/8GxXi0s9qqtkx7RBG8mNpWEZrUhibb9rl9vrQp9D4QGBzyiumUSJzGsd3VXXI8MavE6woeGMw1fH+seIDI8qYbqClG6y3+BOoLlZFspZqAE2TXT+fFXSzw571QCOQ5nWHNxZAiv5DwqqeYrbnIY++Veh4NLhl6vpVfgqqKlUIiEue7JVl+8zA9KHJQqLskT9moip5RO+3elKBpDa0hjcd7cCeeCjYY2oGbKf38OL07bZIMlBNmNVSy7qWPti1bbNOFjczpBPXMamXH2jskzSip/9ummOolAB8duTm6L7T7k+/1tYOz6IXQGgPaP5Xu41FBPIhQfwJFugBen5zhLK8rRzpJ1U= ceph-d88002e2-1120-11ee-9b2b-001a4aab830c 2023-06-22T17:20:35.709 DEBUG:teuthology.orchestra.run.smithi101:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5tQ9nDpBOSN+wtaWGNclkt7PkTanv9ZMInGcJeS0T8zZaeDxvKTlyvYR1jrtDQQxaF24a+cbnEn1sXhp7nu5lZ8smN/uwt2MCOO+yn0B2tCvMwBxmGjHnSxd5AyXscuIUituiObHRDirchwuGlZ3+HWqvlcAI2lXOrA/8GxXi0s9qqtkx7RBG8mNpWEZrUhibb9rl9vrQp9D4QGBzyiumUSJzGsd3VXXI8MavE6woeGMw1fH+seIDI8qYbqClG6y3+BOoLlZFspZqAE2TXT+fFXSzw571QCOQ5nWHNxZAiv5DwqqeYrbnIY++Veh4NLhl6vpVfgqqKlUIiEue7JVl+8zA9KHJQqLskT9moip5RO+3elKBpDa0hjcd7cCeeCjYY2oGbKf38OL07bZIMlBNmNVSy7qWPti1bbNOFjczpBPXMamXH2jskzSip/9ummOolAB8duTm6L7T7k+/1tYOz6IXQGgPaP5Xu41FBPIhQfwJFugBen5zhLK8rRzpJ1U= ceph-d88002e2-1120-11ee-9b2b-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-06-22T17:20:35.751 INFO:teuthology.orchestra.run.smithi101.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5tQ9nDpBOSN+wtaWGNclkt7PkTanv9ZMInGcJeS0T8zZaeDxvKTlyvYR1jrtDQQxaF24a+cbnEn1sXhp7nu5lZ8smN/uwt2MCOO+yn0B2tCvMwBxmGjHnSxd5AyXscuIUituiObHRDirchwuGlZ3+HWqvlcAI2lXOrA/8GxXi0s9qqtkx7RBG8mNpWEZrUhibb9rl9vrQp9D4QGBzyiumUSJzGsd3VXXI8MavE6woeGMw1fH+seIDI8qYbqClG6y3+BOoLlZFspZqAE2TXT+fFXSzw571QCOQ5nWHNxZAiv5DwqqeYrbnIY++Veh4NLhl6vpVfgqqKlUIiEue7JVl+8zA9KHJQqLskT9moip5RO+3elKBpDa0hjcd7cCeeCjYY2oGbKf38OL07bZIMlBNmNVSy7qWPti1bbNOFjczpBPXMamXH2jskzSip/9ummOolAB8duTm6L7T7k+/1tYOz6IXQGgPaP5Xu41FBPIhQfwJFugBen5zhLK8rRzpJ1U= ceph-d88002e2-1120-11ee-9b2b-001a4aab830c 2023-06-22T17:20:35.763 DEBUG:teuthology.orchestra.run.smithi133:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5tQ9nDpBOSN+wtaWGNclkt7PkTanv9ZMInGcJeS0T8zZaeDxvKTlyvYR1jrtDQQxaF24a+cbnEn1sXhp7nu5lZ8smN/uwt2MCOO+yn0B2tCvMwBxmGjHnSxd5AyXscuIUituiObHRDirchwuGlZ3+HWqvlcAI2lXOrA/8GxXi0s9qqtkx7RBG8mNpWEZrUhibb9rl9vrQp9D4QGBzyiumUSJzGsd3VXXI8MavE6woeGMw1fH+seIDI8qYbqClG6y3+BOoLlZFspZqAE2TXT+fFXSzw571QCOQ5nWHNxZAiv5DwqqeYrbnIY++Veh4NLhl6vpVfgqqKlUIiEue7JVl+8zA9KHJQqLskT9moip5RO+3elKBpDa0hjcd7cCeeCjYY2oGbKf38OL07bZIMlBNmNVSy7qWPti1bbNOFjczpBPXMamXH2jskzSip/9ummOolAB8duTm6L7T7k+/1tYOz6IXQGgPaP5Xu41FBPIhQfwJFugBen5zhLK8rRzpJ1U= ceph-d88002e2-1120-11ee-9b2b-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-06-22T17:20:35.805 INFO:teuthology.orchestra.run.smithi133.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5tQ9nDpBOSN+wtaWGNclkt7PkTanv9ZMInGcJeS0T8zZaeDxvKTlyvYR1jrtDQQxaF24a+cbnEn1sXhp7nu5lZ8smN/uwt2MCOO+yn0B2tCvMwBxmGjHnSxd5AyXscuIUituiObHRDirchwuGlZ3+HWqvlcAI2lXOrA/8GxXi0s9qqtkx7RBG8mNpWEZrUhibb9rl9vrQp9D4QGBzyiumUSJzGsd3VXXI8MavE6woeGMw1fH+seIDI8qYbqClG6y3+BOoLlZFspZqAE2TXT+fFXSzw571QCOQ5nWHNxZAiv5DwqqeYrbnIY++Veh4NLhl6vpVfgqqKlUIiEue7JVl+8zA9KHJQqLskT9moip5RO+3elKBpDa0hjcd7cCeeCjYY2oGbKf38OL07bZIMlBNmNVSy7qWPti1bbNOFjczpBPXMamXH2jskzSip/9ummOolAB8duTm6L7T7k+/1tYOz6IXQGgPaP5Xu41FBPIhQfwJFugBen5zhLK8rRzpJ1U= ceph-d88002e2-1120-11ee-9b2b-001a4aab830c 2023-06-22T17:20:35.817 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-06-22T17:20:36.303 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:36 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/259628903' entity='client.admin' 2023-06-22T17:20:36.324 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-22T17:20:37.947 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:37.945+0000 7f1132065700 1 -- 172.21.15.39:0/4253412759 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f112c065890 msgr2=0x7f112c101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:37.947 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:37.945+0000 7f1132065700 1 --2- 172.21.15.39:0/4253412759 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f112c065890 0x7f112c101620 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f111c0048c0 tx=0x7f111c009e30 comp rx=0 tx=0).stop 2023-06-22T17:20:37.947 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:37.945+0000 7f1132065700 1 -- 172.21.15.39:0/4253412759 shutdown_connections 2023-06-22T17:20:37.947 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:37.945+0000 7f1132065700 1 --2- 172.21.15.39:0/4253412759 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f112c065890 0x7f112c101620 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:37.947 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:37.945+0000 7f1132065700 1 -- 172.21.15.39:0/4253412759 >> 172.21.15.39:0/4253412759 conn(0x7f112c0f6c30 msgr2=0x7f112c0f9060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:37.948 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:37.945+0000 7f1132065700 1 -- 172.21.15.39:0/4253412759 shutdown_connections 2023-06-22T17:20:37.948 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:37.945+0000 7f1132065700 1 -- 172.21.15.39:0/4253412759 wait complete. 2023-06-22T17:20:37.948 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:37.945+0000 7f1132065700 1 Processor -- start 2023-06-22T17:20:37.948 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:37.946+0000 7f1132065700 1 -- start start 2023-06-22T17:20:37.948 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:37.946+0000 7f1132065700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f112c065890 0x7f112c10ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:37.948 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:37.946+0000 7f1132065700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f112c10efa0 con 0x7f112c065890 2023-06-22T17:20:37.949 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:37.946+0000 7f112b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f112c065890 0x7f112c10ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:37.949 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:37.946+0000 7f112b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f112c065890 0x7f112c10ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33200/0 (socket says 172.21.15.39:33200) 2023-06-22T17:20:37.949 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:37.946+0000 7f112b7fe700 1 -- 172.21.15.39:0/221137264 learned_addr learned my addr 172.21.15.39:0/221137264 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:37.949 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:37.946+0000 7f112b7fe700 1 -- 172.21.15.39:0/221137264 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f111c005040 con 0x7f112c065890 2023-06-22T17:20:37.949 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:37.946+0000 7f112b7fe700 1 --2- 172.21.15.39:0/221137264 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f112c065890 0x7f112c10ea60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f111c00c010 tx=0x7f111c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:37.950 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:37.947+0000 7f1113fff700 1 -- 172.21.15.39:0/221137264 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f111c00e040 con 0x7f112c065890 2023-06-22T17:20:37.950 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:37.947+0000 7f1113fff700 1 -- 172.21.15.39:0/221137264 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f111c009960 con 0x7f112c065890 2023-06-22T17:20:37.950 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:37.947+0000 7f1132065700 1 -- 172.21.15.39:0/221137264 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f112c10f1a0 con 0x7f112c065890 2023-06-22T17:20:37.951 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:37.947+0000 7f1113fff700 1 -- 172.21.15.39:0/221137264 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f111c01c3f0 con 0x7f112c065890 2023-06-22T17:20:37.951 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:37.947+0000 7f1132065700 1 -- 172.21.15.39:0/221137264 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f112c10fe40 con 0x7f112c065890 2023-06-22T17:20:37.951 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:37.947+0000 7f1113fff700 1 -- 172.21.15.39:0/221137264 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f111c01c670 con 0x7f112c065890 2023-06-22T17:20:37.951 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:37.948+0000 7f1113fff700 1 --2- 172.21.15.39:0/221137264 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f1114036660 0x7f1114038b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:37.951 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:37.948+0000 7f1113fff700 1 -- 172.21.15.39:0/221137264 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f111c04a5b0 con 0x7f112c065890 2023-06-22T17:20:37.951 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:37.948+0000 7f112affd700 1 --2- 172.21.15.39:0/221137264 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f1114036660 0x7f1114038b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:37.951 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:37.948+0000 7f112affd700 1 --2- 172.21.15.39:0/221137264 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f1114036660 0x7f1114038b10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1120009990 tx=0x7f1120006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:37.951 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:37.948+0000 7f1132065700 1 -- 172.21.15.39:0/221137264 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11180052e0 con 0x7f112c065890 2023-06-22T17:20:37.954 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:37.953+0000 7f1113fff700 1 -- 172.21.15.39:0/221137264 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f111c025400 con 0x7f112c065890 2023-06-22T17:20:38.161 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:38.160+0000 7f1132065700 1 -- 172.21.15.39:0/221137264 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f1118005b10 con 0x7f112c065890 2023-06-22T17:20:38.163 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:38.163+0000 7f1113fff700 1 -- 172.21.15.39:0/221137264 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f111c048030 con 0x7f112c065890 2023-06-22T17:20:38.166 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:38.165+0000 7f1132065700 1 -- 172.21.15.39:0/221137264 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f1114036660 msgr2=0x7f1114038b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:38.166 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:38.165+0000 7f1132065700 1 --2- 172.21.15.39:0/221137264 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f1114036660 0x7f1114038b10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1120009990 tx=0x7f1120006e40 comp rx=0 tx=0).stop 2023-06-22T17:20:38.166 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:38.165+0000 7f1132065700 1 -- 172.21.15.39:0/221137264 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f112c065890 msgr2=0x7f112c10ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:38.167 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:38.165+0000 7f1132065700 1 --2- 172.21.15.39:0/221137264 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f112c065890 0x7f112c10ea60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f111c00c010 tx=0x7f111c009e30 comp rx=0 tx=0).stop 2023-06-22T17:20:38.167 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:38.165+0000 7f1132065700 1 -- 172.21.15.39:0/221137264 shutdown_connections 2023-06-22T17:20:38.167 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:38.165+0000 7f1132065700 1 --2- 172.21.15.39:0/221137264 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f1114036660 0x7f1114038b10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:38.167 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:38.165+0000 7f1132065700 1 --2- 172.21.15.39:0/221137264 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f112c065890 0x7f112c10ea60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:38.167 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:38.165+0000 7f1132065700 1 -- 172.21.15.39:0/221137264 >> 172.21.15.39:0/221137264 conn(0x7f112c0f6c30 msgr2=0x7f112c0f78e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:38.167 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:38.166+0000 7f1132065700 1 -- 172.21.15.39:0/221137264 shutdown_connections 2023-06-22T17:20:38.168 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:38.166+0000 7f1132065700 1 -- 172.21.15.39:0/221137264 wait complete. 2023-06-22T17:20:38.688 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-06-22T17:20:38.689 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-06-22T17:20:39.287 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-22T17:20:39.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:39 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/221137264' entity='client.admin' 2023-06-22T17:20:39.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:39 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:39.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:39 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:39.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:39 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi039", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:20:39.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:39 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:39.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:39 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:20:39.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:39 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:40.626 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.625+0000 7f766fdfd700 1 -- 172.21.15.39:0/3098350282 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76680fb570 msgr2=0x7f76680fb980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:40.627 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.625+0000 7f766fdfd700 1 --2- 172.21.15.39:0/3098350282 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76680fb570 0x7f76680fb980 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f76640048c0 tx=0x7f7664009e30 comp rx=0 tx=0).stop 2023-06-22T17:20:40.627 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.625+0000 7f766fdfd700 1 -- 172.21.15.39:0/3098350282 shutdown_connections 2023-06-22T17:20:40.627 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.625+0000 7f766fdfd700 1 --2- 172.21.15.39:0/3098350282 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76680fb570 0x7f76680fb980 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:40.627 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.625+0000 7f766fdfd700 1 -- 172.21.15.39:0/3098350282 >> 172.21.15.39:0/3098350282 conn(0x7f76680f6c30 msgr2=0x7f76680f9060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:40.627 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.625+0000 7f766fdfd700 1 -- 172.21.15.39:0/3098350282 shutdown_connections 2023-06-22T17:20:40.627 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.625+0000 7f766fdfd700 1 -- 172.21.15.39:0/3098350282 wait complete. 2023-06-22T17:20:40.628 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.626+0000 7f766fdfd700 1 Processor -- start 2023-06-22T17:20:40.628 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.626+0000 7f766fdfd700 1 -- start start 2023-06-22T17:20:40.628 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.626+0000 7f766fdfd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76680fb570 0x7f766810b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:40.628 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.626+0000 7f766fdfd700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766810bd40 con 0x7f76680fb570 2023-06-22T17:20:40.628 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.626+0000 7f766db99700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76680fb570 0x7f766810b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:40.628 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.626+0000 7f766db99700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76680fb570 0x7f766810b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33208/0 (socket says 172.21.15.39:33208) 2023-06-22T17:20:40.628 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.626+0000 7f766db99700 1 -- 172.21.15.39:0/3800191856 learned_addr learned my addr 172.21.15.39:0/3800191856 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:40.630 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.626+0000 7f766db99700 1 -- 172.21.15.39:0/3800191856 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7664005040 con 0x7f76680fb570 2023-06-22T17:20:40.630 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.626+0000 7f766db99700 1 --2- 172.21.15.39:0/3800191856 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76680fb570 0x7f766810b800 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f766400c010 tx=0x7f7664009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:40.630 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.626+0000 7f765e7fc700 1 -- 172.21.15.39:0/3800191856 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f766400e040 con 0x7f76680fb570 2023-06-22T17:20:40.630 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.627+0000 7f765e7fc700 1 -- 172.21.15.39:0/3800191856 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f7664009960 con 0x7f76680fb570 2023-06-22T17:20:40.630 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.627+0000 7f766fdfd700 1 -- 172.21.15.39:0/3800191856 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f766810bf40 con 0x7f76680fb570 2023-06-22T17:20:40.630 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.627+0000 7f765e7fc700 1 -- 172.21.15.39:0/3800191856 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f766401c3f0 con 0x7f76680fb570 2023-06-22T17:20:40.631 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.627+0000 7f766fdfd700 1 -- 172.21.15.39:0/3800191856 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f766810cbe0 con 0x7f76680fb570 2023-06-22T17:20:40.631 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.627+0000 7f765e7fc700 1 -- 172.21.15.39:0/3800191856 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f76640136b0 con 0x7f76680fb570 2023-06-22T17:20:40.631 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.628+0000 7f766fdfd700 1 -- 172.21.15.39:0/3800191856 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f766805f060 con 0x7f76680fb570 2023-06-22T17:20:40.631 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.628+0000 7f765e7fc700 1 --2- 172.21.15.39:0/3800191856 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f7654036660 0x7f7654038b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:40.631 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.628+0000 7f765e7fc700 1 -- 172.21.15.39:0/3800191856 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f7664049600 con 0x7f76680fb570 2023-06-22T17:20:40.631 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.628+0000 7f766d398700 1 --2- 172.21.15.39:0/3800191856 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f7654036660 0x7f7654038b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:40.631 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.628+0000 7f766d398700 1 --2- 172.21.15.39:0/3800191856 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f7654036660 0x7f7654038b10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7658009990 tx=0x7f7658006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:40.633 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.633+0000 7f765e7fc700 1 -- 172.21.15.39:0/3800191856 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f7664011550 con 0x7f76680fb570 2023-06-22T17:20:40.832 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.831+0000 7f766fdfd700 1 -- 172.21.15.39:0/3800191856 --> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f766810d760 con 0x7f7654036660 2023-06-22T17:20:40.840 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.839+0000 7f765e7fc700 1 -- 172.21.15.39:0/3800191856 <== mgr.14150 v2:172.21.15.39:6800/886078282 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f766810d760 con 0x7f7654036660 2023-06-22T17:20:40.842 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.841+0000 7f766fdfd700 1 -- 172.21.15.39:0/3800191856 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f7654036660 msgr2=0x7f7654038b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:40.842 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.841+0000 7f766fdfd700 1 --2- 172.21.15.39:0/3800191856 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f7654036660 0x7f7654038b10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7658009990 tx=0x7f7658006e40 comp rx=0 tx=0).stop 2023-06-22T17:20:40.842 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.841+0000 7f766fdfd700 1 -- 172.21.15.39:0/3800191856 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76680fb570 msgr2=0x7f766810b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:40.843 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.841+0000 7f766fdfd700 1 --2- 172.21.15.39:0/3800191856 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76680fb570 0x7f766810b800 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f766400c010 tx=0x7f7664009e30 comp rx=0 tx=0).stop 2023-06-22T17:20:40.843 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.842+0000 7f766fdfd700 1 -- 172.21.15.39:0/3800191856 shutdown_connections 2023-06-22T17:20:40.843 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.842+0000 7f766fdfd700 1 --2- 172.21.15.39:0/3800191856 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f7654036660 0x7f7654038b10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:40.843 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.842+0000 7f766fdfd700 1 --2- 172.21.15.39:0/3800191856 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76680fb570 0x7f766810b800 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:40.843 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.842+0000 7f766fdfd700 1 -- 172.21.15.39:0/3800191856 >> 172.21.15.39:0/3800191856 conn(0x7f76680f6c30 msgr2=0x7f76680f78e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:40.843 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.842+0000 7f766fdfd700 1 -- 172.21.15.39:0/3800191856 shutdown_connections 2023-06-22T17:20:40.843 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:40.842+0000 7f766fdfd700 1 -- 172.21.15.39:0/3800191856 wait complete. 2023-06-22T17:20:41.345 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi101 2023-06-22T17:20:41.346 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-06-22T17:20:41.347 DEBUG:teuthology.orchestra.run.smithi101:> dd of=/etc/ceph/ceph.conf 2023-06-22T17:20:41.365 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-06-22T17:20:41.366 DEBUG:teuthology.orchestra.run.smithi101:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-06-22T17:20:41.423 INFO:tasks.cephadm:Adding host smithi101 to orchestrator... 2023-06-22T17:20:41.424 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph orch host add smithi101 2023-06-22T17:20:42.020 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-22T17:20:42.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:41 smithi039 ceph-mon[84805]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:20:42.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:41 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:42.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:41 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:20:42.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:41 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:20:42.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:41 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:20:42.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:41 smithi039 ceph-mon[84805]: Updating smithi039:/etc/ceph/ceph.conf 2023-06-22T17:20:42.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:41 smithi039 ceph-mon[84805]: Updating smithi039:/var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/config/ceph.conf 2023-06-22T17:20:42.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:41 smithi039 ceph-mon[84805]: Updating smithi039:/etc/ceph/ceph.client.admin.keyring 2023-06-22T17:20:42.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:41 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:42.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:41 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:42.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:41 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:43.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:42 smithi039 ceph-mon[84805]: Updating smithi039:/var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/config/ceph.client.admin.keyring 2023-06-22T17:20:43.228 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:43.227+0000 7fc4742f5700 1 -- 172.21.15.39:0/3211040462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc46c101600 msgr2=0x7fc46c1019d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:43.228 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:43.227+0000 7fc4742f5700 1 --2- 172.21.15.39:0/3211040462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc46c101600 0x7fc46c1019d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc4680048c0 tx=0x7fc468009e30 comp rx=0 tx=0).stop 2023-06-22T17:20:43.228 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:43.227+0000 7fc4742f5700 1 -- 172.21.15.39:0/3211040462 shutdown_connections 2023-06-22T17:20:43.228 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:43.227+0000 7fc4742f5700 1 --2- 172.21.15.39:0/3211040462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc46c101600 0x7fc46c1019d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:43.228 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:43.227+0000 7fc4742f5700 1 -- 172.21.15.39:0/3211040462 >> 172.21.15.39:0/3211040462 conn(0x7fc46c071b80 msgr2=0x7fc46c071f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:43.229 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:43.227+0000 7fc4742f5700 1 -- 172.21.15.39:0/3211040462 shutdown_connections 2023-06-22T17:20:43.229 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:43.227+0000 7fc4742f5700 1 -- 172.21.15.39:0/3211040462 wait complete. 2023-06-22T17:20:43.229 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:43.227+0000 7fc4742f5700 1 Processor -- start 2023-06-22T17:20:43.229 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:43.227+0000 7fc4742f5700 1 -- start start 2023-06-22T17:20:43.230 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:43.228+0000 7fc4742f5700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc46c101600 0x7fc46c0fd260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:43.231 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:43.228+0000 7fc4742f5700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46c0fd7a0 con 0x7fc46c101600 2023-06-22T17:20:43.231 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:43.228+0000 7fc472091700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc46c101600 0x7fc46c0fd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:43.231 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:43.228+0000 7fc472091700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc46c101600 0x7fc46c0fd260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33228/0 (socket says 172.21.15.39:33228) 2023-06-22T17:20:43.231 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:43.228+0000 7fc472091700 1 -- 172.21.15.39:0/3348227248 learned_addr learned my addr 172.21.15.39:0/3348227248 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:43.231 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:43.228+0000 7fc472091700 1 -- 172.21.15.39:0/3348227248 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc468005040 con 0x7fc46c101600 2023-06-22T17:20:43.231 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:43.228+0000 7fc472091700 1 --2- 172.21.15.39:0/3348227248 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc46c101600 0x7fc46c0fd260 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc46800c010 tx=0x7fc468009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:43.231 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:43.228+0000 7fc462ffd700 1 -- 172.21.15.39:0/3348227248 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc46800e040 con 0x7fc46c101600 2023-06-22T17:20:43.231 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:43.228+0000 7fc462ffd700 1 -- 172.21.15.39:0/3348227248 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7fc4680098a0 con 0x7fc46c101600 2023-06-22T17:20:43.232 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:43.228+0000 7fc4742f5700 1 -- 172.21.15.39:0/3348227248 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc46c0fd9a0 con 0x7fc46c101600 2023-06-22T17:20:43.232 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:43.228+0000 7fc462ffd700 1 -- 172.21.15.39:0/3348227248 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc468013e70 con 0x7fc46c101600 2023-06-22T17:20:43.232 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:43.228+0000 7fc4742f5700 1 -- 172.21.15.39:0/3348227248 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc46c0fde10 con 0x7fc46c101600 2023-06-22T17:20:43.232 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:43.229+0000 7fc462ffd700 1 -- 172.21.15.39:0/3348227248 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) v1 ==== 43275+0+0 (secure 0 0 0) 0x7fc4680093c0 con 0x7fc46c101600 2023-06-22T17:20:43.232 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:43.229+0000 7fc4742f5700 1 -- 172.21.15.39:0/3348227248 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc46c05f060 con 0x7fc46c101600 2023-06-22T17:20:43.232 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:43.230+0000 7fc462ffd700 1 --2- 172.21.15.39:0/3348227248 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fc4580366b0 0x7fc458038b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:43.232 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:43.230+0000 7fc462ffd700 1 -- 172.21.15.39:0/3348227248 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fc4680494a0 con 0x7fc46c101600 2023-06-22T17:20:43.233 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:43.230+0000 7fc471890700 1 --2- 172.21.15.39:0/3348227248 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fc4580366b0 0x7fc458038b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:43.233 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:43.230+0000 7fc471890700 1 --2- 172.21.15.39:0/3348227248 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fc4580366b0 0x7fc458038b60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc45c009990 tx=0x7fc45c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:43.235 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:43.234+0000 7fc462ffd700 1 -- 172.21.15.39:0/3348227248 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fc4680119c0 con 0x7fc46c101600 2023-06-22T17:20:43.433 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:43.433+0000 7fc4742f5700 1 -- 172.21.15.39:0/3348227248 --> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi101", "target": ["mon-mgr", ""]}) v1 -- 0x7fc46c0fe9a0 con 0x7fc4580366b0 2023-06-22T17:20:44.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:43 smithi039 ceph-mon[84805]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi101", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:20:45.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:44 smithi039 ceph-mon[84805]: Deploying cephadm binary to smithi101 2023-06-22T17:20:45.272 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:45.272+0000 7fc462ffd700 1 -- 172.21.15.39:0/3348227248 <== mgr.14150 v2:172.21.15.39:6800/886078282 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fc46c0fe9a0 con 0x7fc4580366b0 2023-06-22T17:20:45.273 INFO:teuthology.orchestra.run.smithi039.stdout:Added host 'smithi101' with addr '172.21.15.101' 2023-06-22T17:20:45.275 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:45.274+0000 7fc4742f5700 1 -- 172.21.15.39:0/3348227248 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fc4580366b0 msgr2=0x7fc458038b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:45.275 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:45.274+0000 7fc4742f5700 1 --2- 172.21.15.39:0/3348227248 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fc4580366b0 0x7fc458038b60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc45c009990 tx=0x7fc45c006e40 comp rx=0 tx=0).stop 2023-06-22T17:20:45.275 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:45.274+0000 7fc4742f5700 1 -- 172.21.15.39:0/3348227248 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc46c101600 msgr2=0x7fc46c0fd260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:45.275 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:45.274+0000 7fc4742f5700 1 --2- 172.21.15.39:0/3348227248 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc46c101600 0x7fc46c0fd260 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc46800c010 tx=0x7fc468009e30 comp rx=0 tx=0).stop 2023-06-22T17:20:45.275 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:45.274+0000 7fc4742f5700 1 -- 172.21.15.39:0/3348227248 shutdown_connections 2023-06-22T17:20:45.275 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:45.274+0000 7fc4742f5700 1 --2- 172.21.15.39:0/3348227248 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fc4580366b0 0x7fc458038b60 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:45.276 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:45.274+0000 7fc4742f5700 1 --2- 172.21.15.39:0/3348227248 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc46c101600 0x7fc46c0fd260 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:45.276 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:45.274+0000 7fc4742f5700 1 -- 172.21.15.39:0/3348227248 >> 172.21.15.39:0/3348227248 conn(0x7fc46c071b80 msgr2=0x7fc46c0fa950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:45.276 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:45.274+0000 7fc4742f5700 1 -- 172.21.15.39:0/3348227248 shutdown_connections 2023-06-22T17:20:45.276 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:45.274+0000 7fc4742f5700 1 -- 172.21.15.39:0/3348227248 wait complete. 2023-06-22T17:20:45.661 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph orch host ls --format=json 2023-06-22T17:20:46.256 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-22T17:20:46.640 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:46 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:46.640 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:46 smithi039 ceph-mon[84805]: Added host smithi101 2023-06-22T17:20:46.640 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:46 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:20:46.641 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:46 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:47.477 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.476+0000 7f1f709a3700 1 -- 172.21.15.39:0/3320331875 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f6c1026a0 msgr2=0x7f1f6c102a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:47.477 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.476+0000 7f1f709a3700 1 --2- 172.21.15.39:0/3320331875 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f6c1026a0 0x7f1f6c102a70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c0048c0 tx=0x7f1f5c009e30 comp rx=0 tx=0).stop 2023-06-22T17:20:47.477 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.476+0000 7f1f709a3700 1 -- 172.21.15.39:0/3320331875 shutdown_connections 2023-06-22T17:20:47.478 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.476+0000 7f1f709a3700 1 --2- 172.21.15.39:0/3320331875 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f6c1026a0 0x7f1f6c102a70 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:47.478 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.476+0000 7f1f709a3700 1 -- 172.21.15.39:0/3320331875 >> 172.21.15.39:0/3320331875 conn(0x7f1f6c0f8200 msgr2=0x7f1f6c0fa610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:47.478 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.476+0000 7f1f709a3700 1 -- 172.21.15.39:0/3320331875 shutdown_connections 2023-06-22T17:20:47.478 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.476+0000 7f1f709a3700 1 -- 172.21.15.39:0/3320331875 wait complete. 2023-06-22T17:20:47.478 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.476+0000 7f1f709a3700 1 Processor -- start 2023-06-22T17:20:47.478 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.476+0000 7f1f709a3700 1 -- start start 2023-06-22T17:20:47.478 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.476+0000 7f1f709a3700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f6c1026a0 0x7f1f6c112960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:47.479 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.476+0000 7f1f709a3700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6c112ea0 con 0x7f1f6c1026a0 2023-06-22T17:20:47.479 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.477+0000 7f1f6a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f6c1026a0 0x7f1f6c112960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:47.479 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.477+0000 7f1f6a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f6c1026a0 0x7f1f6c112960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:42014/0 (socket says 172.21.15.39:42014) 2023-06-22T17:20:47.479 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.477+0000 7f1f6a59c700 1 -- 172.21.15.39:0/3490389540 learned_addr learned my addr 172.21.15.39:0/3490389540 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:47.479 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.477+0000 7f1f6a59c700 1 -- 172.21.15.39:0/3490389540 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f5c005040 con 0x7f1f6c1026a0 2023-06-22T17:20:47.480 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.477+0000 7f1f6a59c700 1 --2- 172.21.15.39:0/3490389540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f6c1026a0 0x7f1f6c112960 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c00c010 tx=0x7f1f5c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:47.480 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.477+0000 7f1f5affd700 1 -- 172.21.15.39:0/3490389540 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f5c00e040 con 0x7f1f6c1026a0 2023-06-22T17:20:47.481 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.477+0000 7f1f709a3700 1 -- 172.21.15.39:0/3490389540 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f6c1130a0 con 0x7f1f6c1026a0 2023-06-22T17:20:47.481 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.477+0000 7f1f5affd700 1 -- 172.21.15.39:0/3490389540 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f1f5c0098a0 con 0x7f1f6c1026a0 2023-06-22T17:20:47.481 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.477+0000 7f1f5affd700 1 -- 172.21.15.39:0/3490389540 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f5c013e70 con 0x7f1f6c1026a0 2023-06-22T17:20:47.481 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.477+0000 7f1f709a3700 1 -- 172.21.15.39:0/3490389540 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f6c10c820 con 0x7f1f6c1026a0 2023-06-22T17:20:47.481 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.478+0000 7f1f5affd700 1 -- 172.21.15.39:0/3490389540 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f1f5c0136b0 con 0x7f1f6c1026a0 2023-06-22T17:20:47.481 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.478+0000 7f1f709a3700 1 -- 172.21.15.39:0/3490389540 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f6c05f060 con 0x7f1f6c1026a0 2023-06-22T17:20:47.481 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.478+0000 7f1f5affd700 1 --2- 172.21.15.39:0/3490389540 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f1f54036660 0x7f1f54038b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:47.481 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.478+0000 7f1f5affd700 1 -- 172.21.15.39:0/3490389540 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f1f5c049510 con 0x7f1f6c1026a0 2023-06-22T17:20:47.482 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.478+0000 7f1f69d9b700 1 --2- 172.21.15.39:0/3490389540 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f1f54036660 0x7f1f54038b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:47.482 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.479+0000 7f1f69d9b700 1 --2- 172.21.15.39:0/3490389540 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f1f54036660 0x7f1f54038b10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1f60009990 tx=0x7f1f60006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:47.483 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.483+0000 7f1f5affd700 1 -- 172.21.15.39:0/3490389540 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f1f5c011960 con 0x7f1f6c1026a0 2023-06-22T17:20:47.680 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.679+0000 7f1f709a3700 1 -- 172.21.15.39:0/3490389540 --> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1f6c04b9e0 con 0x7f1f54036660 2023-06-22T17:20:47.681 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.681+0000 7f1f5affd700 1 -- 172.21.15.39:0/3490389540 <== mgr.14150 v2:172.21.15.39:6800/886078282 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f1f6c04b9e0 con 0x7f1f54036660 2023-06-22T17:20:47.682 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:20:47.682 INFO:teuthology.orchestra.run.smithi039.stdout:[{"addr": "172.21.15.39", "hostname": "smithi039", "labels": [], "status": ""}, {"addr": "172.21.15.101", "hostname": "smithi101", "labels": [], "status": ""}] 2023-06-22T17:20:47.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:47 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:47.683 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.683+0000 7f1f709a3700 1 -- 172.21.15.39:0/3490389540 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f1f54036660 msgr2=0x7f1f54038b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:47.684 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.683+0000 7f1f709a3700 1 --2- 172.21.15.39:0/3490389540 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f1f54036660 0x7f1f54038b10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1f60009990 tx=0x7f1f60006e40 comp rx=0 tx=0).stop 2023-06-22T17:20:47.684 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.683+0000 7f1f709a3700 1 -- 172.21.15.39:0/3490389540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f6c1026a0 msgr2=0x7f1f6c112960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:47.684 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.683+0000 7f1f709a3700 1 --2- 172.21.15.39:0/3490389540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f6c1026a0 0x7f1f6c112960 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c00c010 tx=0x7f1f5c009e30 comp rx=0 tx=0).stop 2023-06-22T17:20:47.684 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.683+0000 7f1f709a3700 1 -- 172.21.15.39:0/3490389540 shutdown_connections 2023-06-22T17:20:47.684 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.683+0000 7f1f709a3700 1 --2- 172.21.15.39:0/3490389540 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f1f54036660 0x7f1f54038b10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:47.684 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.683+0000 7f1f709a3700 1 --2- 172.21.15.39:0/3490389540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f6c1026a0 0x7f1f6c112960 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:47.685 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.683+0000 7f1f709a3700 1 -- 172.21.15.39:0/3490389540 >> 172.21.15.39:0/3490389540 conn(0x7f1f6c0f8200 msgr2=0x7f1f6c0f8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:47.685 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.683+0000 7f1f709a3700 1 -- 172.21.15.39:0/3490389540 shutdown_connections 2023-06-22T17:20:47.685 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:47.683+0000 7f1f709a3700 1 -- 172.21.15.39:0/3490389540 wait complete. 2023-06-22T17:20:48.208 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi133 2023-06-22T17:20:48.209 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-06-22T17:20:48.209 DEBUG:teuthology.orchestra.run.smithi133:> dd of=/etc/ceph/ceph.conf 2023-06-22T17:20:48.226 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-06-22T17:20:48.227 DEBUG:teuthology.orchestra.run.smithi133:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-06-22T17:20:48.283 INFO:tasks.cephadm:Adding host smithi133 to orchestrator... 2023-06-22T17:20:48.284 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph orch host add smithi133 2023-06-22T17:20:48.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:48 smithi039 ceph-mon[84805]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:20:48.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:48 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:48.881 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-22T17:20:49.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:49 smithi039 ceph-mon[84805]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-06-22T17:20:50.094 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:50.093+0000 7f6fa1c1d700 1 -- 172.21.15.39:0/2831286740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f9c1026a0 msgr2=0x7f6f9c102a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:50.094 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:50.093+0000 7f6fa1c1d700 1 --2- 172.21.15.39:0/2831286740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f9c1026a0 0x7f6f9c102a70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6f940048c0 tx=0x7f6f94009e30 comp rx=0 tx=0).stop 2023-06-22T17:20:50.094 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:50.093+0000 7f6fa1c1d700 1 -- 172.21.15.39:0/2831286740 shutdown_connections 2023-06-22T17:20:50.095 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:50.093+0000 7f6fa1c1d700 1 --2- 172.21.15.39:0/2831286740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f9c1026a0 0x7f6f9c102a70 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:50.095 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:50.093+0000 7f6fa1c1d700 1 -- 172.21.15.39:0/2831286740 >> 172.21.15.39:0/2831286740 conn(0x7f6f9c0f8200 msgr2=0x7f6f9c0fa610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:50.095 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:50.093+0000 7f6fa1c1d700 1 -- 172.21.15.39:0/2831286740 shutdown_connections 2023-06-22T17:20:50.095 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:50.093+0000 7f6fa1c1d700 1 -- 172.21.15.39:0/2831286740 wait complete. 2023-06-22T17:20:50.095 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:50.093+0000 7f6fa1c1d700 1 Processor -- start 2023-06-22T17:20:50.095 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:50.094+0000 7f6fa1c1d700 1 -- start start 2023-06-22T17:20:50.097 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:50.094+0000 7f6fa1c1d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f9c1026a0 0x7f6f9c06fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:50.097 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:50.094+0000 7f6fa1c1d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9c070270 con 0x7f6f9c1026a0 2023-06-22T17:20:50.097 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:50.094+0000 7f6f9b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f9c1026a0 0x7f6f9c06fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:50.097 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:50.094+0000 7f6f9b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f9c1026a0 0x7f6f9c06fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:42022/0 (socket says 172.21.15.39:42022) 2023-06-22T17:20:50.097 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:50.094+0000 7f6f9b7fe700 1 -- 172.21.15.39:0/1917734319 learned_addr learned my addr 172.21.15.39:0/1917734319 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:50.098 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:50.094+0000 7f6f9b7fe700 1 -- 172.21.15.39:0/1917734319 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f94005040 con 0x7f6f9c1026a0 2023-06-22T17:20:50.098 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:50.094+0000 7f6f9b7fe700 1 --2- 172.21.15.39:0/1917734319 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f9c1026a0 0x7f6f9c06fd30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6f94008f00 tx=0x7f6f94009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:50.098 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:50.095+0000 7f6f93fff700 1 -- 172.21.15.39:0/1917734319 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6f9400e040 con 0x7f6f9c1026a0 2023-06-22T17:20:50.098 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:50.095+0000 7f6fa1c1d700 1 -- 172.21.15.39:0/1917734319 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f9c070470 con 0x7f6f9c1026a0 2023-06-22T17:20:50.098 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:50.095+0000 7f6f93fff700 1 -- 172.21.15.39:0/1917734319 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f6f94009850 con 0x7f6f9c1026a0 2023-06-22T17:20:50.098 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:50.095+0000 7f6f93fff700 1 -- 172.21.15.39:0/1917734319 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6f94013db0 con 0x7f6f9c1026a0 2023-06-22T17:20:50.098 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:50.095+0000 7f6fa1c1d700 1 -- 172.21.15.39:0/1917734319 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f9c0708e0 con 0x7f6f9c1026a0 2023-06-22T17:20:50.098 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:50.095+0000 7f6f93fff700 1 -- 172.21.15.39:0/1917734319 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f6f9401c440 con 0x7f6f9c1026a0 2023-06-22T17:20:50.099 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:50.096+0000 7f6f93fff700 1 --2- 172.21.15.39:0/1917734319 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f6f88036660 0x7f6f88038b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:50.099 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:50.096+0000 7f6fa1c1d700 1 -- 172.21.15.39:0/1917734319 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f9c101910 con 0x7f6f9c1026a0 2023-06-22T17:20:50.099 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:50.096+0000 7f6f93fff700 1 -- 172.21.15.39:0/1917734319 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6f940494c0 con 0x7f6f9c1026a0 2023-06-22T17:20:50.099 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:50.096+0000 7f6f9affd700 1 --2- 172.21.15.39:0/1917734319 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f6f88036660 0x7f6f88038b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:50.099 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:50.096+0000 7f6f9affd700 1 --2- 172.21.15.39:0/1917734319 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f6f88036660 0x7f6f88038b10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6f84009990 tx=0x7f6f84006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:50.101 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:50.100+0000 7f6f93fff700 1 -- 172.21.15.39:0/1917734319 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f6f94011960 con 0x7f6f9c1026a0 2023-06-22T17:20:50.301 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:50.301+0000 7f6fa1c1d700 1 -- 172.21.15.39:0/1917734319 --> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi133", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f9c04b970 con 0x7f6f88036660 2023-06-22T17:20:50.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:50 smithi039 ceph-mon[84805]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:20:50.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:50 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:50.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:50 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:50.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:50 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:50.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:50 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:50.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:50 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi101", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:20:50.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:50 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:20:50.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:50 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:20:51.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:51 smithi039 ceph-mon[84805]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi133", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:20:51.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:51 smithi039 ceph-mon[84805]: Updating smithi101:/etc/ceph/ceph.conf 2023-06-22T17:20:51.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:51 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:51.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:51 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:51.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:51 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:52.136 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:52.135+0000 7f6f93fff700 1 -- 172.21.15.39:0/1917734319 <== mgr.14150 v2:172.21.15.39:6800/886078282 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f6f9c04b970 con 0x7f6f88036660 2023-06-22T17:20:52.136 INFO:teuthology.orchestra.run.smithi039.stdout:Added host 'smithi133' with addr '172.21.15.133' 2023-06-22T17:20:52.138 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:52.137+0000 7f6fa1c1d700 1 -- 172.21.15.39:0/1917734319 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f6f88036660 msgr2=0x7f6f88038b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:52.139 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:52.137+0000 7f6fa1c1d700 1 --2- 172.21.15.39:0/1917734319 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f6f88036660 0x7f6f88038b10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6f84009990 tx=0x7f6f84006e40 comp rx=0 tx=0).stop 2023-06-22T17:20:52.139 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:52.137+0000 7f6fa1c1d700 1 -- 172.21.15.39:0/1917734319 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f9c1026a0 msgr2=0x7f6f9c06fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:52.139 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:52.137+0000 7f6fa1c1d700 1 --2- 172.21.15.39:0/1917734319 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f9c1026a0 0x7f6f9c06fd30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6f94008f00 tx=0x7f6f94009e30 comp rx=0 tx=0).stop 2023-06-22T17:20:52.139 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:52.137+0000 7f6fa1c1d700 1 -- 172.21.15.39:0/1917734319 shutdown_connections 2023-06-22T17:20:52.139 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:52.137+0000 7f6fa1c1d700 1 --2- 172.21.15.39:0/1917734319 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f6f88036660 0x7f6f88038b10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:52.139 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:52.137+0000 7f6fa1c1d700 1 --2- 172.21.15.39:0/1917734319 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f9c1026a0 0x7f6f9c06fd30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:52.139 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:52.137+0000 7f6fa1c1d700 1 -- 172.21.15.39:0/1917734319 >> 172.21.15.39:0/1917734319 conn(0x7f6f9c0f8200 msgr2=0x7f6f9c0f9fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:52.140 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:52.138+0000 7f6fa1c1d700 1 -- 172.21.15.39:0/1917734319 shutdown_connections 2023-06-22T17:20:52.140 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:52.138+0000 7f6fa1c1d700 1 -- 172.21.15.39:0/1917734319 wait complete. 2023-06-22T17:20:52.522 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph orch host ls --format=json 2023-06-22T17:20:52.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:52 smithi039 ceph-mon[84805]: Updating smithi101:/var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/config/ceph.conf 2023-06-22T17:20:52.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:52 smithi039 ceph-mon[84805]: Updating smithi101:/etc/ceph/ceph.client.admin.keyring 2023-06-22T17:20:52.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:52 smithi039 ceph-mon[84805]: Deploying cephadm binary to smithi133 2023-06-22T17:20:52.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:52 smithi039 ceph-mon[84805]: Updating smithi101:/var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/config/ceph.client.admin.keyring 2023-06-22T17:20:52.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:52 smithi039 ceph-mon[84805]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:20:52.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:52 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:52.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:52 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:20:53.114 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-22T17:20:53.678 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:53 smithi039 ceph-mon[84805]: Added host smithi133 2023-06-22T17:20:53.679 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:53 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:53.679 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:53 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:54.350 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.348+0000 7f6de3d1f700 1 -- 172.21.15.39:0/2900893838 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ddc0fd680 msgr2=0x7f6ddc0fda50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:54.350 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.348+0000 7f6de3d1f700 1 --2- 172.21.15.39:0/2900893838 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ddc0fd680 0x7f6ddc0fda50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6dd80048c0 tx=0x7f6dd8009e30 comp rx=0 tx=0).stop 2023-06-22T17:20:54.350 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.349+0000 7f6de3d1f700 1 -- 172.21.15.39:0/2900893838 shutdown_connections 2023-06-22T17:20:54.350 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.349+0000 7f6de3d1f700 1 --2- 172.21.15.39:0/2900893838 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ddc0fd680 0x7f6ddc0fda50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:54.351 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.349+0000 7f6de3d1f700 1 -- 172.21.15.39:0/2900893838 >> 172.21.15.39:0/2900893838 conn(0x7f6ddc0713d0 msgr2=0x7f6ddc0717d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:54.351 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.349+0000 7f6de3d1f700 1 -- 172.21.15.39:0/2900893838 shutdown_connections 2023-06-22T17:20:54.351 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.349+0000 7f6de3d1f700 1 -- 172.21.15.39:0/2900893838 wait complete. 2023-06-22T17:20:54.351 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.349+0000 7f6de3d1f700 1 Processor -- start 2023-06-22T17:20:54.351 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.349+0000 7f6de3d1f700 1 -- start start 2023-06-22T17:20:54.353 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.350+0000 7f6de3d1f700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ddc0fd680 0x7f6ddc112960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:54.353 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.350+0000 7f6de3d1f700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ddc112ea0 con 0x7f6ddc0fd680 2023-06-22T17:20:54.353 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.350+0000 7f6de1abb700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ddc0fd680 0x7f6ddc112960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:54.353 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.350+0000 7f6de1abb700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ddc0fd680 0x7f6ddc112960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:42052/0 (socket says 172.21.15.39:42052) 2023-06-22T17:20:54.353 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.350+0000 7f6de1abb700 1 -- 172.21.15.39:0/1822699421 learned_addr learned my addr 172.21.15.39:0/1822699421 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:54.353 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.350+0000 7f6de1abb700 1 -- 172.21.15.39:0/1822699421 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd8005040 con 0x7f6ddc0fd680 2023-06-22T17:20:54.354 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.350+0000 7f6de1abb700 1 --2- 172.21.15.39:0/1822699421 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ddc0fd680 0x7f6ddc112960 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6dd8008f00 tx=0x7f6dd8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:54.354 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.350+0000 7f6dd27fc700 1 -- 172.21.15.39:0/1822699421 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6dd800e040 con 0x7f6ddc0fd680 2023-06-22T17:20:54.354 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.350+0000 7f6de3d1f700 1 -- 172.21.15.39:0/1822699421 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ddc1130a0 con 0x7f6ddc0fd680 2023-06-22T17:20:54.354 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.350+0000 7f6dd27fc700 1 -- 172.21.15.39:0/1822699421 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f6dd8009850 con 0x7f6ddc0fd680 2023-06-22T17:20:54.354 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.350+0000 7f6dd27fc700 1 -- 172.21.15.39:0/1822699421 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6dd8013db0 con 0x7f6ddc0fd680 2023-06-22T17:20:54.354 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.350+0000 7f6de3d1f700 1 -- 172.21.15.39:0/1822699421 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ddc10c820 con 0x7f6ddc0fd680 2023-06-22T17:20:54.354 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.351+0000 7f6dd27fc700 1 -- 172.21.15.39:0/1822699421 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f6dd801c440 con 0x7f6ddc0fd680 2023-06-22T17:20:54.355 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.351+0000 7f6dd27fc700 1 --2- 172.21.15.39:0/1822699421 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f6dc8036660 0x7f6dc8038b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:54.355 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.351+0000 7f6dd27fc700 1 -- 172.21.15.39:0/1822699421 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6dd80494f0 con 0x7f6ddc0fd680 2023-06-22T17:20:54.355 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.351+0000 7f6de12ba700 1 --2- 172.21.15.39:0/1822699421 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f6dc8036660 0x7f6dc8038b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:54.355 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.351+0000 7f6de3d1f700 1 -- 172.21.15.39:0/1822699421 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ddc05f060 con 0x7f6ddc0fd680 2023-06-22T17:20:54.355 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.352+0000 7f6de12ba700 1 --2- 172.21.15.39:0/1822699421 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f6dc8036660 0x7f6dc8038b10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc009990 tx=0x7f6dcc006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:54.357 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.356+0000 7f6dd27fc700 1 -- 172.21.15.39:0/1822699421 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f6dd8011960 con 0x7f6ddc0fd680 2023-06-22T17:20:54.559 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.559+0000 7f6de3d1f700 1 -- 172.21.15.39:0/1822699421 --> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6ddc04b970 con 0x7f6dc8036660 2023-06-22T17:20:54.560 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.560+0000 7f6dd27fc700 1 -- 172.21.15.39:0/1822699421 <== mgr.14150 v2:172.21.15.39:6800/886078282 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+240 (secure 0 0 0) 0x7f6ddc04b970 con 0x7f6dc8036660 2023-06-22T17:20:54.560 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:20:54.561 INFO:teuthology.orchestra.run.smithi039.stdout:[{"addr": "172.21.15.39", "hostname": "smithi039", "labels": [], "status": ""}, {"addr": "172.21.15.101", "hostname": "smithi101", "labels": [], "status": ""}, {"addr": "172.21.15.133", "hostname": "smithi133", "labels": [], "status": ""}] 2023-06-22T17:20:54.562 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.562+0000 7f6de3d1f700 1 -- 172.21.15.39:0/1822699421 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f6dc8036660 msgr2=0x7f6dc8038b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:54.562 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.562+0000 7f6de3d1f700 1 --2- 172.21.15.39:0/1822699421 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f6dc8036660 0x7f6dc8038b10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc009990 tx=0x7f6dcc006e40 comp rx=0 tx=0).stop 2023-06-22T17:20:54.563 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.562+0000 7f6de3d1f700 1 -- 172.21.15.39:0/1822699421 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ddc0fd680 msgr2=0x7f6ddc112960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:54.563 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.562+0000 7f6de3d1f700 1 --2- 172.21.15.39:0/1822699421 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ddc0fd680 0x7f6ddc112960 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6dd8008f00 tx=0x7f6dd8009e30 comp rx=0 tx=0).stop 2023-06-22T17:20:54.563 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.562+0000 7f6de3d1f700 1 -- 172.21.15.39:0/1822699421 shutdown_connections 2023-06-22T17:20:54.563 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.562+0000 7f6de3d1f700 1 --2- 172.21.15.39:0/1822699421 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f6dc8036660 0x7f6dc8038b10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:54.563 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.562+0000 7f6de3d1f700 1 --2- 172.21.15.39:0/1822699421 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ddc0fd680 0x7f6ddc112960 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:54.563 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.562+0000 7f6de3d1f700 1 -- 172.21.15.39:0/1822699421 >> 172.21.15.39:0/1822699421 conn(0x7f6ddc0713d0 msgr2=0x7f6ddc100f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:54.563 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.562+0000 7f6de3d1f700 1 -- 172.21.15.39:0/1822699421 shutdown_connections 2023-06-22T17:20:54.563 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:54.562+0000 7f6de3d1f700 1 -- 172.21.15.39:0/1822699421 wait complete. 2023-06-22T17:20:54.770 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:54 smithi039 ceph-mon[84805]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:20:54.964 INFO:tasks.cephadm:Setting crush tunables to default 2023-06-22T17:20:54.965 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph osd crush tunables default 2023-06-22T17:20:55.557 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-22T17:20:55.851 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:55 smithi039 ceph-mon[84805]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-06-22T17:20:55.852 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:55 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:55.852 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:55 smithi039 ceph-mon[84805]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:20:56.781 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:56.779+0000 7fcd8cc7f700 1 -- 172.21.15.39:0/3680849465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd881026a0 msgr2=0x7fcd88102a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:56.781 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:56.779+0000 7fcd8cc7f700 1 --2- 172.21.15.39:0/3680849465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd881026a0 0x7fcd88102a70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcd780048c0 tx=0x7fcd78009e30 comp rx=0 tx=0).stop 2023-06-22T17:20:56.781 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:56.779+0000 7fcd8cc7f700 1 -- 172.21.15.39:0/3680849465 shutdown_connections 2023-06-22T17:20:56.781 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:56.779+0000 7fcd8cc7f700 1 --2- 172.21.15.39:0/3680849465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd881026a0 0x7fcd88102a70 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:56.781 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:56.779+0000 7fcd8cc7f700 1 -- 172.21.15.39:0/3680849465 >> 172.21.15.39:0/3680849465 conn(0x7fcd880f8200 msgr2=0x7fcd880fa610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:56.782 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:56.779+0000 7fcd8cc7f700 1 -- 172.21.15.39:0/3680849465 shutdown_connections 2023-06-22T17:20:56.782 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:56.779+0000 7fcd8cc7f700 1 -- 172.21.15.39:0/3680849465 wait complete. 2023-06-22T17:20:56.782 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:56.779+0000 7fcd8cc7f700 1 Processor -- start 2023-06-22T17:20:56.782 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:56.779+0000 7fcd8cc7f700 1 -- start start 2023-06-22T17:20:56.782 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:56.780+0000 7fcd8cc7f700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd881026a0 0x7fcd88112a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:56.782 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:56.780+0000 7fcd8cc7f700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd88112f40 con 0x7fcd881026a0 2023-06-22T17:20:56.782 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:56.780+0000 7fcd8659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd881026a0 0x7fcd88112a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:56.782 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:56.780+0000 7fcd8659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd881026a0 0x7fcd88112a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:60092/0 (socket says 172.21.15.39:60092) 2023-06-22T17:20:56.783 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:56.780+0000 7fcd8659c700 1 -- 172.21.15.39:0/2576570957 learned_addr learned my addr 172.21.15.39:0/2576570957 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:20:56.783 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:56.780+0000 7fcd8659c700 1 -- 172.21.15.39:0/2576570957 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd78005040 con 0x7fcd881026a0 2023-06-22T17:20:56.783 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:56.780+0000 7fcd8659c700 1 --2- 172.21.15.39:0/2576570957 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd881026a0 0x7fcd88112a00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcd7800c010 tx=0x7fcd78009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:56.783 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:56.780+0000 7fcd76ffd700 1 -- 172.21.15.39:0/2576570957 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd7800e040 con 0x7fcd881026a0 2023-06-22T17:20:56.783 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:56.780+0000 7fcd8cc7f700 1 -- 172.21.15.39:0/2576570957 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd88113140 con 0x7fcd881026a0 2023-06-22T17:20:56.783 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:56.780+0000 7fcd76ffd700 1 -- 172.21.15.39:0/2576570957 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7fcd780098a0 con 0x7fcd881026a0 2023-06-22T17:20:56.783 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:56.780+0000 7fcd76ffd700 1 -- 172.21.15.39:0/2576570957 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd78013e70 con 0x7fcd881026a0 2023-06-22T17:20:56.784 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:56.780+0000 7fcd8cc7f700 1 -- 172.21.15.39:0/2576570957 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd881135b0 con 0x7fcd881026a0 2023-06-22T17:20:56.784 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:56.781+0000 7fcd76ffd700 1 -- 172.21.15.39:0/2576570957 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) v1 ==== 43275+0+0 (secure 0 0 0) 0x7fcd7801c480 con 0x7fcd881026a0 2023-06-22T17:20:56.784 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:56.781+0000 7fcd76ffd700 1 --2- 172.21.15.39:0/2576570957 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fcd70036660 0x7fcd70038b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:56.785 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:56.781+0000 7fcd76ffd700 1 -- 172.21.15.39:0/2576570957 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fcd780494f0 con 0x7fcd881026a0 2023-06-22T17:20:56.785 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:56.781+0000 7fcd85d9b700 1 --2- 172.21.15.39:0/2576570957 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fcd70036660 0x7fcd70038b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:56.785 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:56.781+0000 7fcd8cc7f700 1 -- 172.21.15.39:0/2576570957 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd8805f060 con 0x7fcd881026a0 2023-06-22T17:20:56.785 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:56.782+0000 7fcd85d9b700 1 --2- 172.21.15.39:0/2576570957 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fcd70036660 0x7fcd70038b10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c009990 tx=0x7fcd7c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:56.787 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:56.786+0000 7fcd76ffd700 1 -- 172.21.15.39:0/2576570957 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fcd78011960 con 0x7fcd881026a0 2023-06-22T17:20:56.989 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:56.988+0000 7fcd8cc7f700 1 -- 172.21.15.39:0/2576570957 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fcd8804b970 con 0x7fcd881026a0 2023-06-22T17:20:57.051 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:57.051+0000 7fcd76ffd700 1 -- 172.21.15.39:0/2576570957 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fcd78011370 con 0x7fcd881026a0 2023-06-22T17:20:57.054 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:57.053+0000 7fcd8cc7f700 1 -- 172.21.15.39:0/2576570957 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fcd70036660 msgr2=0x7fcd70038b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:57.054 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:57.053+0000 7fcd8cc7f700 1 --2- 172.21.15.39:0/2576570957 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fcd70036660 0x7fcd70038b10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c009990 tx=0x7fcd7c006e40 comp rx=0 tx=0).stop 2023-06-22T17:20:57.054 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:57.053+0000 7fcd8cc7f700 1 -- 172.21.15.39:0/2576570957 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd881026a0 msgr2=0x7fcd88112a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:57.054 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:57.053+0000 7fcd8cc7f700 1 --2- 172.21.15.39:0/2576570957 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd881026a0 0x7fcd88112a00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcd7800c010 tx=0x7fcd78009e30 comp rx=0 tx=0).stop 2023-06-22T17:20:57.054 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:57.053+0000 7fcd8cc7f700 1 -- 172.21.15.39:0/2576570957 shutdown_connections 2023-06-22T17:20:57.054 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:57.053+0000 7fcd8cc7f700 1 --2- 172.21.15.39:0/2576570957 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fcd70036660 0x7fcd70038b10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:57.055 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:57.053+0000 7fcd8cc7f700 1 --2- 172.21.15.39:0/2576570957 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd881026a0 0x7fcd88112a00 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:57.055 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:57.053+0000 7fcd8cc7f700 1 -- 172.21.15.39:0/2576570957 >> 172.21.15.39:0/2576570957 conn(0x7fcd880f8200 msgr2=0x7fcd880f8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:57.055 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:57.053+0000 7fcd8cc7f700 1 -- 172.21.15.39:0/2576570957 shutdown_connections 2023-06-22T17:20:57.055 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:20:57.053+0000 7fcd8cc7f700 1 -- 172.21.15.39:0/2576570957 wait complete. 2023-06-22T17:20:57.056 INFO:teuthology.orchestra.run.smithi039.stderr:adjusted tunables profile to default 2023-06-22T17:20:57.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:57 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/2576570957' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-06-22T17:20:57.498 INFO:tasks.cephadm:Adding mon.a on smithi039 2023-06-22T17:20:57.499 INFO:tasks.cephadm:Adding mon.b on smithi101 2023-06-22T17:20:57.499 INFO:tasks.cephadm:Adding mon.c on smithi133 2023-06-22T17:20:57.499 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph orch apply mon '3;smithi039:172.21.15.39=a;smithi101:172.21.15.101=b;smithi133:172.21.15.133=c' 2023-06-22T17:20:58.122 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/config/ceph.conf 2023-06-22T17:20:58.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:58 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/2576570957' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-06-22T17:20:58.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:58 smithi039 ceph-mon[84805]: osdmap e4: 0 total, 0 up, 0 in 2023-06-22T17:20:58.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:58 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:58.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:58 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:58.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:58 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:58.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:58 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:58.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:58 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi133", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:20:58.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:58 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:20:58.435 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:58 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:20:58.435 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:58 smithi039 ceph-mon[84805]: Updating smithi133:/etc/ceph/ceph.conf 2023-06-22T17:20:58.435 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:58 smithi039 ceph-mon[84805]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:20:58.435 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:58 smithi039 ceph-mon[84805]: Updating smithi133:/var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/config/ceph.conf 2023-06-22T17:20:58.435 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:58 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:58.435 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:58 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:58.436 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:58 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:20:59.323 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.323+0000 7f8529c8d700 1 -- 172.21.15.133:0/122557606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85241026c0 msgr2=0x7f8524102a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:59.324 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.323+0000 7f8529c8d700 1 --2- 172.21.15.133:0/122557606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85241026c0 0x7f8524102a90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f85140048c0 tx=0x7f8514009e30 comp rx=0 tx=0).stop 2023-06-22T17:20:59.324 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.323+0000 7f8529c8d700 1 -- 172.21.15.133:0/122557606 shutdown_connections 2023-06-22T17:20:59.324 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.323+0000 7f8529c8d700 1 --2- 172.21.15.133:0/122557606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85241026c0 0x7f8524102a90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:59.324 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.323+0000 7f8529c8d700 1 -- 172.21.15.133:0/122557606 >> 172.21.15.133:0/122557606 conn(0x7f85240f8200 msgr2=0x7f85240fa610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:59.324 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.323+0000 7f8529c8d700 1 -- 172.21.15.133:0/122557606 shutdown_connections 2023-06-22T17:20:59.324 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.323+0000 7f8529c8d700 1 -- 172.21.15.133:0/122557606 wait complete. 2023-06-22T17:20:59.325 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.324+0000 7f8529c8d700 1 Processor -- start 2023-06-22T17:20:59.326 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.324+0000 7f8529c8d700 1 -- start start 2023-06-22T17:20:59.326 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.324+0000 7f8529c8d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85241026c0 0x7f852406fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:59.326 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.324+0000 7f8529c8d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8524070270 con 0x7f85241026c0 2023-06-22T17:20:59.326 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.324+0000 7f85237fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85241026c0 0x7f852406fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:59.327 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.324+0000 7f85237fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85241026c0 0x7f852406fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.133:32946/0 (socket says 172.21.15.133:32946) 2023-06-22T17:20:59.327 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.324+0000 7f85237fe700 1 -- 172.21.15.133:0/1532718727 learned_addr learned my addr 172.21.15.133:0/1532718727 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-06-22T17:20:59.327 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.325+0000 7f85237fe700 1 -- 172.21.15.133:0/1532718727 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8514005040 con 0x7f85241026c0 2023-06-22T17:20:59.327 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.325+0000 7f85237fe700 1 --2- 172.21.15.133:0/1532718727 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85241026c0 0x7f852406fd30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8514008f00 tx=0x7f8514009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:59.327 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.325+0000 7f850bfff700 1 -- 172.21.15.133:0/1532718727 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f851400e040 con 0x7f85241026c0 2023-06-22T17:20:59.327 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.325+0000 7f8529c8d700 1 -- 172.21.15.133:0/1532718727 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8524070470 con 0x7f85241026c0 2023-06-22T17:20:59.327 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.325+0000 7f850bfff700 1 -- 172.21.15.133:0/1532718727 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f8514009850 con 0x7f85241026c0 2023-06-22T17:20:59.327 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.325+0000 7f850bfff700 1 -- 172.21.15.133:0/1532718727 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8514013db0 con 0x7f85241026c0 2023-06-22T17:20:59.328 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.325+0000 7f8529c8d700 1 -- 172.21.15.133:0/1532718727 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8524070960 con 0x7f85241026c0 2023-06-22T17:20:59.328 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.326+0000 7f8529c8d700 1 -- 172.21.15.133:0/1532718727 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8524070ba0 con 0x7f85241026c0 2023-06-22T17:20:59.328 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.326+0000 7f850bfff700 1 -- 172.21.15.133:0/1532718727 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f8514013640 con 0x7f85241026c0 2023-06-22T17:20:59.328 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.326+0000 7f850bfff700 1 --2- 172.21.15.133:0/1532718727 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f850c036660 0x7f850c038b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:20:59.328 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.326+0000 7f850bfff700 1 -- 172.21.15.133:0/1532718727 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f8514049500 con 0x7f85241026c0 2023-06-22T17:20:59.329 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.327+0000 7f8522ffd700 1 --2- 172.21.15.133:0/1532718727 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f850c036660 0x7f850c038b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:20:59.329 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.327+0000 7f8522ffd700 1 --2- 172.21.15.133:0/1532718727 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f850c036660 0x7f850c038b10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8518009990 tx=0x7f8518006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:20:59.331 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.331+0000 7f850bfff700 1 -- 172.21.15.133:0/1532718727 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f8524070ba0 con 0x7f85241026c0 2023-06-22T17:20:59.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:59 smithi039 ceph-mon[84805]: Updating smithi133:/etc/ceph/ceph.client.admin.keyring 2023-06-22T17:20:59.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:20:59 smithi039 ceph-mon[84805]: Updating smithi133:/var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/config/ceph.client.admin.keyring 2023-06-22T17:20:59.522 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.522+0000 7f8529c8d700 1 -- 172.21.15.133:0/1532718727 --> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi039:172.21.15.39=a;smithi101:172.21.15.101=b;smithi133:172.21.15.133=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f852404b970 con 0x7f850c036660 2023-06-22T17:20:59.529 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.529+0000 7f850bfff700 1 -- 172.21.15.133:0/1532718727 <== mgr.14150 v2:172.21.15.39:6800/886078282 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f852404b970 con 0x7f850c036660 2023-06-22T17:20:59.530 INFO:teuthology.orchestra.run.smithi133.stdout:Scheduled mon update... 2023-06-22T17:20:59.532 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.531+0000 7f8529c8d700 1 -- 172.21.15.133:0/1532718727 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f850c036660 msgr2=0x7f850c038b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:59.532 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.531+0000 7f8529c8d700 1 --2- 172.21.15.133:0/1532718727 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f850c036660 0x7f850c038b10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8518009990 tx=0x7f8518006e40 comp rx=0 tx=0).stop 2023-06-22T17:20:59.532 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.531+0000 7f8529c8d700 1 -- 172.21.15.133:0/1532718727 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85241026c0 msgr2=0x7f852406fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:20:59.532 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.531+0000 7f8529c8d700 1 --2- 172.21.15.133:0/1532718727 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85241026c0 0x7f852406fd30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8514008f00 tx=0x7f8514009e30 comp rx=0 tx=0).stop 2023-06-22T17:20:59.533 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.531+0000 7f8529c8d700 1 -- 172.21.15.133:0/1532718727 shutdown_connections 2023-06-22T17:20:59.533 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.531+0000 7f8529c8d700 1 --2- 172.21.15.133:0/1532718727 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f850c036660 0x7f850c038b10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:59.533 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.531+0000 7f8529c8d700 1 --2- 172.21.15.133:0/1532718727 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85241026c0 0x7f852406fd30 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:20:59.533 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.531+0000 7f8529c8d700 1 -- 172.21.15.133:0/1532718727 >> 172.21.15.133:0/1532718727 conn(0x7f85240f8200 msgr2=0x7f85240fa020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:20:59.533 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.532+0000 7f8529c8d700 1 -- 172.21.15.133:0/1532718727 shutdown_connections 2023-06-22T17:20:59.533 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:20:59.532+0000 7f8529c8d700 1 -- 172.21.15.133:0/1532718727 wait complete. 2023-06-22T17:20:59.990 DEBUG:teuthology.orchestra.run.smithi101:mon.b> sudo journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mon.b.service 2023-06-22T17:20:59.994 DEBUG:teuthology.orchestra.run.smithi133:mon.c> sudo journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mon.c.service 2023-06-22T17:20:59.996 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-06-22T17:20:59.997 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph mon dump -f json 2023-06-22T17:21:00.022 INFO:journalctl@ceph.mon.b.smithi101.stdout:-- Logs begin at Thu 2023-06-22 17:06:34 UTC. -- 2023-06-22T17:21:00.023 INFO:journalctl@ceph.mon.c.smithi133.stdout:-- Logs begin at Thu 2023-06-22 17:06:34 UTC. -- 2023-06-22T17:21:00.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:00 smithi039 ceph-mon[84805]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:00.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:00 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:00.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:00 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:21:00.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:00 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:00.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:00 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:21:00.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:00 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:00.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:00 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-06-22T17:21:00.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:00 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:00.950 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/config/ceph.conf 2023-06-22T17:21:01.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:01 smithi039 ceph-mon[84805]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi039:172.21.15.39=a;smithi101:172.21.15.101=b;smithi133:172.21.15.133=c", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:21:01.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:01 smithi039 ceph-mon[84805]: Saving service mon spec with placement smithi039:172.21.15.39=a;smithi101:172.21.15.101=b;smithi133:172.21.15.133=c;count:3 2023-06-22T17:21:01.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:01 smithi039 ceph-mon[84805]: Deploying daemon mon.c on smithi133 2023-06-22T17:21:02.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:02 smithi039 ceph-mon[84805]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:02.618 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.617+0000 7f559f918700 1 -- 172.21.15.133:0/1536709938 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5598101560 msgr2=0x7f5598101930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:21:02.619 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.617+0000 7f559f918700 1 --2- 172.21.15.133:0/1536709938 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5598101560 0x7f5598101930 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f558c0048c0 tx=0x7f558c009e30 comp rx=0 tx=0).stop 2023-06-22T17:21:02.619 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.617+0000 7f559f918700 1 -- 172.21.15.133:0/1536709938 shutdown_connections 2023-06-22T17:21:02.619 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.617+0000 7f559f918700 1 --2- 172.21.15.133:0/1536709938 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5598101560 0x7f5598101930 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:02.619 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.617+0000 7f559f918700 1 -- 172.21.15.133:0/1536709938 >> 172.21.15.133:0/1536709938 conn(0x7f55980f7100 msgr2=0x7f55980f9510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:21:02.619 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.618+0000 7f559f918700 1 -- 172.21.15.133:0/1536709938 shutdown_connections 2023-06-22T17:21:02.620 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.618+0000 7f559f918700 1 -- 172.21.15.133:0/1536709938 wait complete. 2023-06-22T17:21:02.620 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.618+0000 7f559f918700 1 Processor -- start 2023-06-22T17:21:02.620 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.618+0000 7f559f918700 1 -- start start 2023-06-22T17:21:02.620 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.618+0000 7f559f918700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5598101560 0x7f559810da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:21:02.620 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.618+0000 7f559f918700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559810df80 con 0x7f5598101560 2023-06-22T17:21:02.620 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.619+0000 7f559d6b4700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5598101560 0x7f559810da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:21:02.620 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.619+0000 7f559d6b4700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5598101560 0x7f559810da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.133:32964/0 (socket says 172.21.15.133:32964) 2023-06-22T17:21:02.621 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.619+0000 7f559d6b4700 1 -- 172.21.15.133:0/96149126 learned_addr learned my addr 172.21.15.133:0/96149126 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-06-22T17:21:02.622 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.619+0000 7f559d6b4700 1 -- 172.21.15.133:0/96149126 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f558c005040 con 0x7f5598101560 2023-06-22T17:21:02.622 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.619+0000 7f559d6b4700 1 --2- 172.21.15.133:0/96149126 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5598101560 0x7f559810da40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f558c008f00 tx=0x7f558c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:21:02.622 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.619+0000 7f5589ffb700 1 -- 172.21.15.133:0/96149126 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f558c00e040 con 0x7f5598101560 2023-06-22T17:21:02.622 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.620+0000 7f5589ffb700 1 -- 172.21.15.133:0/96149126 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f558c013420 con 0x7f5598101560 2023-06-22T17:21:02.622 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.620+0000 7f559f918700 1 -- 172.21.15.133:0/96149126 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f559810e180 con 0x7f5598101560 2023-06-22T17:21:02.622 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.620+0000 7f5589ffb700 1 -- 172.21.15.133:0/96149126 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f558c01c570 con 0x7f5598101560 2023-06-22T17:21:02.623 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.620+0000 7f559f918700 1 -- 172.21.15.133:0/96149126 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f559810e5f0 con 0x7f5598101560 2023-06-22T17:21:02.623 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.621+0000 7f5589ffb700 1 -- 172.21.15.133:0/96149126 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f558c01c7e0 con 0x7f5598101560 2023-06-22T17:21:02.623 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.621+0000 7f559f918700 1 -- 172.21.15.133:0/96149126 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5598066430 con 0x7f5598101560 2023-06-22T17:21:02.623 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.621+0000 7f5589ffb700 1 --2- 172.21.15.133:0/96149126 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f5584036690 0x7f5584038b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:21:02.623 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.621+0000 7f5589ffb700 1 -- 172.21.15.133:0/96149126 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f558c0494f0 con 0x7f5598101560 2023-06-22T17:21:02.623 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.621+0000 7f559ceb3700 1 --2- 172.21.15.133:0/96149126 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f5584036690 0x7f5584038b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:21:02.623 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.622+0000 7f559ceb3700 1 --2- 172.21.15.133:0/96149126 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f5584036690 0x7f5584038b40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5594009990 tx=0x7f5594006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:21:02.626 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.626+0000 7f5589ffb700 1 -- 172.21.15.133:0/96149126 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f558c023300 con 0x7f5598101560 2023-06-22T17:21:02.909 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.909+0000 7f559f918700 1 -- 172.21.15.133:0/96149126 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5598100820 con 0x7f5598101560 2023-06-22T17:21:02.910 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.910+0000 7f5589ffb700 1 -- 172.21.15.133:0/96149126 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+731 (secure 0 0 0) 0x7f558c011b60 con 0x7f5598101560 2023-06-22T17:21:02.910 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-06-22T17:21:02.910 INFO:teuthology.orchestra.run.smithi133.stdout:{"epoch":1,"fsid":"d88002e2-1120-11ee-9b2b-001a4aab830c","modified":"2023-06-22T17:19:34.058140Z","created":"2023-06-22T17:19:34.058140Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-06-22T17:21:02.912 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.912+0000 7f559f918700 1 -- 172.21.15.133:0/96149126 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f5584036690 msgr2=0x7f5584038b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:21:02.912 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.912+0000 7f559f918700 1 --2- 172.21.15.133:0/96149126 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f5584036690 0x7f5584038b40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5594009990 tx=0x7f5594006e40 comp rx=0 tx=0).stop 2023-06-22T17:21:02.913 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.912+0000 7f559f918700 1 -- 172.21.15.133:0/96149126 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5598101560 msgr2=0x7f559810da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:21:02.913 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.912+0000 7f559f918700 1 --2- 172.21.15.133:0/96149126 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5598101560 0x7f559810da40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f558c008f00 tx=0x7f558c009e30 comp rx=0 tx=0).stop 2023-06-22T17:21:02.913 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.912+0000 7f559f918700 1 -- 172.21.15.133:0/96149126 shutdown_connections 2023-06-22T17:21:02.913 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.912+0000 7f559f918700 1 --2- 172.21.15.133:0/96149126 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f5584036690 0x7f5584038b40 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:02.913 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.912+0000 7f559f918700 1 --2- 172.21.15.133:0/96149126 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5598101560 0x7f559810da40 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:02.913 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.912+0000 7f559f918700 1 -- 172.21.15.133:0/96149126 >> 172.21.15.133:0/96149126 conn(0x7f55980f7100 msgr2=0x7f55980f7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:21:02.913 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.912+0000 7f559f918700 1 -- 172.21.15.133:0/96149126 shutdown_connections 2023-06-22T17:21:02.914 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:02.912+0000 7f559f918700 1 -- 172.21.15.133:0/96149126 wait complete. 2023-06-22T17:21:02.914 INFO:teuthology.orchestra.run.smithi133.stderr:dumped monmap epoch 1 2023-06-22T17:21:03.326 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 systemd[1]: Started Ceph mon.c for d88002e2-1120-11ee-9b2b-001a4aab830c. 2023-06-22T17:21:03.326 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: mon.c@-1(synchronizing).mds e1 new map 2023-06-22T17:21:03.326 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: mon.c@-1(synchronizing).mds e1 print_map 2023-06-22T17:21:03.326 INFO:journalctl@ceph.mon.c.smithi133.stdout: e1 2023-06-22T17:21:03.327 INFO:journalctl@ceph.mon.c.smithi133.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-06-22T17:21:03.327 INFO:journalctl@ceph.mon.c.smithi133.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-06-22T17:21:03.327 INFO:journalctl@ceph.mon.c.smithi133.stdout: legacy client fscid: -1 2023-06-22T17:21:03.327 INFO:journalctl@ceph.mon.c.smithi133.stdout: 2023-06-22T17:21:03.327 INFO:journalctl@ceph.mon.c.smithi133.stdout: No filesystems configured 2023-06-22T17:21:03.327 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-06-22T17:21:03.327 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-06-22T17:21:03.328 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-06-22T17:21:03.328 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-06-22T17:21:03.328 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-06-22T17:21:03.328 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-06-22T17:21:03.328 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-06-22T17:21:03.328 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-06-22T17:21:03.328 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-06-22T17:21:03.329 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-06-22T17:21:03.329 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: Updating smithi101:/var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/config/ceph.conf 2023-06-22T17:21:03.329 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: Updating smithi101:/etc/ceph/ceph.client.admin.keyring 2023-06-22T17:21:03.329 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: Deploying cephadm binary to smithi133 2023-06-22T17:21:03.330 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: Updating smithi101:/var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/config/ceph.client.admin.keyring 2023-06-22T17:21:03.330 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:03.330 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:03.330 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:21:03.330 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: Added host smithi133 2023-06-22T17:21:03.330 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:03.331 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:03.331 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:03.331 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-06-22T17:21:03.331 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:03.331 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:03.331 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/2576570957' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-06-22T17:21:03.331 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/2576570957' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-06-22T17:21:03.332 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: osdmap e4: 0 total, 0 up, 0 in 2023-06-22T17:21:03.332 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:03.332 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:03.332 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:03.332 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:03.332 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi133", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:21:03.332 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:03.332 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:21:03.333 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: Updating smithi133:/etc/ceph/ceph.conf 2023-06-22T17:21:03.333 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:03.333 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: Updating smithi133:/var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/config/ceph.conf 2023-06-22T17:21:03.333 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:03.334 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:03.334 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:03.334 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: Updating smithi133:/etc/ceph/ceph.client.admin.keyring 2023-06-22T17:21:03.334 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: Updating smithi133:/var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/config/ceph.client.admin.keyring 2023-06-22T17:21:03.334 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:03.334 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:03.334 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:21:03.335 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:03.335 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:21:03.335 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:03.335 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-06-22T17:21:03.335 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:03.335 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi039:172.21.15.39=a;smithi101:172.21.15.101=b;smithi133:172.21.15.133=c", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:21:03.335 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: Saving service mon spec with placement smithi039:172.21.15.39=a;smithi101:172.21.15.101=b;smithi133:172.21.15.133=c;count:3 2023-06-22T17:21:03.335 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: Deploying daemon mon.c on smithi133 2023-06-22T17:21:03.336 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:03.336 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-06-22T17:21:03.336 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:02 smithi133 ceph-mon[86268]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-06-22T17:21:04.587 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-06-22T17:21:04.588 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph mon dump -f json 2023-06-22T17:21:05.175 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.c/config 2023-06-22T17:21:08.419 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:08 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-22T17:21:08.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:08 smithi133 ceph-mon[86268]: mon.a calling monitor election 2023-06-22T17:21:08.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:08 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-22T17:21:08.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:08 smithi133 ceph-mon[86268]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:08.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:08 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-22T17:21:08.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:08 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-22T17:21:08.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:08 smithi133 ceph-mon[86268]: mon.c calling monitor election 2023-06-22T17:21:08.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:08 smithi133 ceph-mon[86268]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:08.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:08 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-22T17:21:08.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:08 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-22T17:21:08.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:08 smithi133 ceph-mon[86268]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:08.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:08 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-22T17:21:08.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:08 smithi133 ceph-mon[86268]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-06-22T17:21:08.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:08 smithi133 ceph-mon[86268]: monmap e2: 2 mons at {a=[v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0],c=[v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0]} removed_ranks: {} 2023-06-22T17:21:08.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:08 smithi133 ceph-mon[86268]: fsmap 2023-06-22T17:21:08.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:08 smithi133 ceph-mon[86268]: osdmap e4: 0 total, 0 up, 0 in 2023-06-22T17:21:08.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:08 smithi133 ceph-mon[86268]: mgrmap e13: x(active, since 40s) 2023-06-22T17:21:08.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:08 smithi133 ceph-mon[86268]: overall HEALTH_OK 2023-06-22T17:21:08.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:08 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:08.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:08 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:08.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:08 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-22T17:21:08.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:08 smithi039 ceph-mon[84805]: mon.a calling monitor election 2023-06-22T17:21:08.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:08 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-22T17:21:08.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:08 smithi039 ceph-mon[84805]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:08.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:08 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-22T17:21:08.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:08 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-22T17:21:08.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:08 smithi039 ceph-mon[84805]: mon.c calling monitor election 2023-06-22T17:21:08.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:08 smithi039 ceph-mon[84805]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:08.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:08 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-22T17:21:08.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:08 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-22T17:21:08.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:08 smithi039 ceph-mon[84805]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:08.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:08 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-22T17:21:08.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:08 smithi039 ceph-mon[84805]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-06-22T17:21:08.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:08 smithi039 ceph-mon[84805]: monmap e2: 2 mons at {a=[v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0],c=[v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0]} removed_ranks: {} 2023-06-22T17:21:08.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:08 smithi039 ceph-mon[84805]: fsmap 2023-06-22T17:21:08.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:08 smithi039 ceph-mon[84805]: osdmap e4: 0 total, 0 up, 0 in 2023-06-22T17:21:08.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:08 smithi039 ceph-mon[84805]: mgrmap e13: x(active, since 40s) 2023-06-22T17:21:08.435 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:08 smithi039 ceph-mon[84805]: overall HEALTH_OK 2023-06-22T17:21:08.435 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:08 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:08.435 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:08 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:09.260 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.259+0000 7f8351eca700 1 -- 172.21.15.133:0/2633725681 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8334003580 msgr2=0x7f8334005a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:21:09.260 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.259+0000 7f8351eca700 1 --2- 172.21.15.133:0/2633725681 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8334003580 0x7f8334005a00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f834c0fc6b0 tx=0x7f833c00b040 comp rx=0 tx=0).stop 2023-06-22T17:21:09.260 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.259+0000 7f8351eca700 1 -- 172.21.15.133:0/2633725681 shutdown_connections 2023-06-22T17:21:09.261 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.259+0000 7f8351eca700 1 --2- 172.21.15.133:0/2633725681 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8334003580 0x7f8334005a00 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:09.261 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.259+0000 7f8351eca700 1 -- 172.21.15.133:0/2633725681 >> 172.21.15.133:0/2633725681 conn(0x7f834c0f8300 msgr2=0x7f834c0fa710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:21:09.261 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.259+0000 7f8351eca700 1 -- 172.21.15.133:0/2633725681 shutdown_connections 2023-06-22T17:21:09.261 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.259+0000 7f8351eca700 1 -- 172.21.15.133:0/2633725681 wait complete. 2023-06-22T17:21:09.261 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.260+0000 7f8351eca700 1 Processor -- start 2023-06-22T17:21:09.263 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.261+0000 7f8351eca700 1 -- start start 2023-06-22T17:21:09.263 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.261+0000 7f8351eca700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8334003580 0x7f834c112bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:21:09.263 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.261+0000 7f8351eca700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f834c113100 0x7f834c10ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:21:09.263 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.261+0000 7f8351eca700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834c113630 con 0x7f8334003580 2023-06-22T17:21:09.264 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.261+0000 7f8351eca700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834c1137a0 con 0x7f834c113100 2023-06-22T17:21:09.264 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.261+0000 7f834affd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f834c113100 0x7f834c10ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:21:09.264 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.261+0000 7f834affd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f834c113100 0x7f834c10ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:52480/0 (socket says 172.21.15.133:52480) 2023-06-22T17:21:09.264 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.261+0000 7f834affd700 1 -- 172.21.15.133:0/1022419216 learned_addr learned my addr 172.21.15.133:0/1022419216 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-06-22T17:21:09.264 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.261+0000 7f834b7fe700 1 --2- 172.21.15.133:0/1022419216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8334003580 0x7f834c112bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:21:09.264 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.261+0000 7f834affd700 1 -- 172.21.15.133:0/1022419216 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f834c113100 msgr2=0x7f834c10ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-06-22T17:21:09.264 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.261+0000 7f834affd700 1 -- 172.21.15.133:0/1022419216 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f834c113100 msgr2=0x7f834c10ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-06-22T17:21:09.264 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.261+0000 7f834affd700 1 --2- 172.21.15.133:0/1022419216 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f834c113100 0x7f834c10ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-06-22T17:21:09.265 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.261+0000 7f834affd700 1 --2- 172.21.15.133:0/1022419216 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f834c113100 0x7f834c10ca80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T17:21:09.265 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.261+0000 7f834b7fe700 1 -- 172.21.15.133:0/1022419216 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f834c113100 msgr2=0x7f834c10ca80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-06-22T17:21:09.265 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.261+0000 7f834b7fe700 1 --2- 172.21.15.133:0/1022419216 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f834c113100 0x7f834c10ca80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:09.265 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.261+0000 7f834b7fe700 1 -- 172.21.15.133:0/1022419216 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833c005040 con 0x7f8334003580 2023-06-22T17:21:09.265 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.262+0000 7f834b7fe700 1 --2- 172.21.15.133:0/1022419216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8334003580 0x7f834c112bc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f833c0040d0 tx=0x7f833c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:21:09.265 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.262+0000 7f8333fff700 1 -- 172.21.15.133:0/1022419216 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f833c01b070 con 0x7f8334003580 2023-06-22T17:21:09.265 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.262+0000 7f8333fff700 1 -- 172.21.15.133:0/1022419216 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f833c020b30 con 0x7f8334003580 2023-06-22T17:21:09.266 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.262+0000 7f8351eca700 1 -- 172.21.15.133:0/1022419216 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f834c10d080 con 0x7f8334003580 2023-06-22T17:21:09.266 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.262+0000 7f8333fff700 1 -- 172.21.15.133:0/1022419216 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f833c01f9a0 con 0x7f8334003580 2023-06-22T17:21:09.266 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.262+0000 7f8351eca700 1 -- 172.21.15.133:0/1022419216 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f834c10d5c0 con 0x7f8334003580 2023-06-22T17:21:09.266 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.263+0000 7f8333fff700 1 -- 172.21.15.133:0/1022419216 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f833c01fc60 con 0x7f8334003580 2023-06-22T17:21:09.266 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.263+0000 7f8333fff700 1 --2- 172.21.15.133:0/1022419216 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f8338036790 0x7f8338038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:21:09.266 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.263+0000 7f8351eca700 1 -- 172.21.15.133:0/1022419216 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f834c05f060 con 0x7f8334003580 2023-06-22T17:21:09.266 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.263+0000 7f8333fff700 1 -- 172.21.15.133:0/1022419216 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f833c04a630 con 0x7f8334003580 2023-06-22T17:21:09.267 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.263+0000 7f834affd700 1 --2- 172.21.15.133:0/1022419216 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f8338036790 0x7f8338038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:21:09.267 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.264+0000 7f834affd700 1 --2- 172.21.15.133:0/1022419216 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f8338036790 0x7f8338038c40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8340006fd0 tx=0x7f8340008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:21:09.268 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.269+0000 7f8333fff700 1 -- 172.21.15.133:0/1022419216 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f833c01ead0 con 0x7f8334003580 2023-06-22T17:21:09.419 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:09 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:09.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:09 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-06-22T17:21:09.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:09 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:09.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:09 smithi133 ceph-mon[86268]: Deploying daemon mon.b on smithi101 2023-06-22T17:21:09.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:09 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-22T17:21:09.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:09 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:09.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:09 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-06-22T17:21:09.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:09 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:09.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:09 smithi039 ceph-mon[84805]: Deploying daemon mon.b on smithi101 2023-06-22T17:21:09.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:09 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-22T17:21:09.530 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.530+0000 7f8351eca700 1 -- 172.21.15.133:0/1022419216 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f834c04ba00 con 0x7f8334003580 2023-06-22T17:21:09.530 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.531+0000 7f8333fff700 1 -- 172.21.15.133:0/1022419216 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1001 (secure 0 0 0) 0x7f833c031570 con 0x7f8334003580 2023-06-22T17:21:09.531 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-06-22T17:21:09.531 INFO:teuthology.orchestra.run.smithi133.stdout:{"epoch":2,"fsid":"d88002e2-1120-11ee-9b2b-001a4aab830c","modified":"2023-06-22T17:21:02.997774Z","created":"2023-06-22T17:19:34.058140Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-06-22T17:21:09.533 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.532+0000 7f8351eca700 1 -- 172.21.15.133:0/1022419216 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f8338036790 msgr2=0x7f8338038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:21:09.533 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.532+0000 7f8351eca700 1 --2- 172.21.15.133:0/1022419216 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f8338036790 0x7f8338038c40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8340006fd0 tx=0x7f8340008040 comp rx=0 tx=0).stop 2023-06-22T17:21:09.533 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.532+0000 7f8351eca700 1 -- 172.21.15.133:0/1022419216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8334003580 msgr2=0x7f834c112bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:21:09.533 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.532+0000 7f8351eca700 1 --2- 172.21.15.133:0/1022419216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8334003580 0x7f834c112bc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f833c0040d0 tx=0x7f833c00b040 comp rx=0 tx=0).stop 2023-06-22T17:21:09.533 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.532+0000 7f8351eca700 1 -- 172.21.15.133:0/1022419216 shutdown_connections 2023-06-22T17:21:09.534 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.532+0000 7f8351eca700 1 --2- 172.21.15.133:0/1022419216 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f8338036790 0x7f8338038c40 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:09.534 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.532+0000 7f8351eca700 1 --2- 172.21.15.133:0/1022419216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8334003580 0x7f834c112bc0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:09.534 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.532+0000 7f8351eca700 1 --2- 172.21.15.133:0/1022419216 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f834c113100 0x7f834c10ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:09.534 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.532+0000 7f8351eca700 1 -- 172.21.15.133:0/1022419216 >> 172.21.15.133:0/1022419216 conn(0x7f834c0f8300 msgr2=0x7f834c105910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:21:09.534 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.533+0000 7f8351eca700 1 -- 172.21.15.133:0/1022419216 shutdown_connections 2023-06-22T17:21:09.534 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:09.533+0000 7f8351eca700 1 -- 172.21.15.133:0/1022419216 wait complete. 2023-06-22T17:21:09.535 INFO:teuthology.orchestra.run.smithi133.stderr:dumped monmap epoch 2 2023-06-22T17:21:10.419 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:10 smithi133 ceph-mon[86268]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:10.419 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:10 smithi133 ceph-mon[86268]: from='client.? 172.21.15.133:0/1022419216' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-06-22T17:21:10.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:10 smithi039 ceph-mon[84805]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:10.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:10 smithi039 ceph-mon[84805]: from='client.? 172.21.15.133:0/1022419216' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-06-22T17:21:10.875 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:10 smithi101 systemd[1]: Starting Ceph mon.b for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:21:10.922 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-06-22T17:21:10.922 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph mon dump -f json 2023-06-22T17:21:11.504 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.c/config 2023-06-22T17:21:11.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: set uid:gid to 167:167 (ceph:ceph) 2023-06-22T17:21:11.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: ceph version 17.2.6-719-g42ddc46e (42ddc46e6e6582d8e29a5a982a81886fe028be2c) quincy (stable), process ceph-mon, pid 8 2023-06-22T17:21:11.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: pidfile_write: ignore empty --pid-file 2023-06-22T17:21:11.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: load: jerasure load: lrc 2023-06-22T17:21:11.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: RocksDB version: 6.15.5 2023-06-22T17:21:11.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-06-22T17:21:11.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Compile date Jun 21 2023 2023-06-22T17:21:11.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: DB SUMMARY 2023-06-22T17:21:11.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: DB Session ID: V5JYHCUM82SCDRYU8BLK 2023-06-22T17:21:11.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: CURRENT file: CURRENT 2023-06-22T17:21:11.628 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: IDENTITY file: IDENTITY 2023-06-22T17:21:11.628 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2023-06-22T17:21:11.628 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2023-06-22T17:21:11.628 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2023-06-22T17:21:11.628 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.error_if_exists: 0 2023-06-22T17:21:11.629 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.create_if_missing: 0 2023-06-22T17:21:11.629 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.paranoid_checks: 1 2023-06-22T17:21:11.629 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-06-22T17:21:11.630 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.env: 0x55ecca173a80 2023-06-22T17:21:11.630 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.fs: Posix File System 2023-06-22T17:21:11.630 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.info_log: 0x55eccbd35da0 2023-06-22T17:21:11.631 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.max_file_opening_threads: 16 2023-06-22T17:21:11.631 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.statistics: (nil) 2023-06-22T17:21:11.631 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.use_fsync: 0 2023-06-22T17:21:11.632 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.max_log_file_size: 0 2023-06-22T17:21:11.632 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-06-22T17:21:11.632 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.log_file_time_to_roll: 0 2023-06-22T17:21:11.632 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.keep_log_file_num: 1000 2023-06-22T17:21:11.633 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.recycle_log_file_num: 0 2023-06-22T17:21:11.633 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.allow_fallocate: 1 2023-06-22T17:21:11.633 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.allow_mmap_reads: 0 2023-06-22T17:21:11.634 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.allow_mmap_writes: 0 2023-06-22T17:21:11.634 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.use_direct_reads: 0 2023-06-22T17:21:11.634 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-06-22T17:21:11.634 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.create_missing_column_families: 0 2023-06-22T17:21:11.635 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.db_log_dir: 2023-06-22T17:21:11.635 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2023-06-22T17:21:11.635 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.table_cache_numshardbits: 6 2023-06-22T17:21:11.635 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.WAL_ttl_seconds: 0 2023-06-22T17:21:11.636 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.WAL_size_limit_MB: 0 2023-06-22T17:21:11.636 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-06-22T17:21:11.636 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-06-22T17:21:11.636 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.is_fd_close_on_exec: 1 2023-06-22T17:21:11.637 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.advise_random_on_open: 1 2023-06-22T17:21:11.637 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.db_write_buffer_size: 0 2023-06-22T17:21:11.637 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.write_buffer_manager: 0x55eccbdfa8d0 2023-06-22T17:21:11.638 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-06-22T17:21:11.638 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-06-22T17:21:11.638 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-06-22T17:21:11.638 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.use_adaptive_mutex: 0 2023-06-22T17:21:11.639 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.rate_limiter: (nil) 2023-06-22T17:21:11.639 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-06-22T17:21:11.639 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.wal_recovery_mode: 2 2023-06-22T17:21:11.639 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.enable_thread_tracking: 0 2023-06-22T17:21:11.640 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.enable_pipelined_write: 0 2023-06-22T17:21:11.640 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.unordered_write: 0 2023-06-22T17:21:11.640 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-06-22T17:21:11.641 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-06-22T17:21:11.641 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-06-22T17:21:11.641 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-06-22T17:21:11.641 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.row_cache: None 2023-06-22T17:21:11.642 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.wal_filter: None 2023-06-22T17:21:11.642 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-06-22T17:21:11.642 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.allow_ingest_behind: 0 2023-06-22T17:21:11.642 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.preserve_deletes: 0 2023-06-22T17:21:11.643 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.two_write_queues: 0 2023-06-22T17:21:11.643 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.manual_wal_flush: 0 2023-06-22T17:21:11.643 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.atomic_flush: 0 2023-06-22T17:21:11.644 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-06-22T17:21:11.644 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.persist_stats_to_disk: 0 2023-06-22T17:21:11.644 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.write_dbid_to_manifest: 0 2023-06-22T17:21:11.644 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.log_readahead_size: 0 2023-06-22T17:21:11.645 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-06-22T17:21:11.645 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.best_efforts_recovery: 0 2023-06-22T17:21:11.645 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-06-22T17:21:11.646 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-06-22T17:21:11.646 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.allow_data_in_errors: 0 2023-06-22T17:21:11.646 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.db_host_id: __hostname__ 2023-06-22T17:21:11.646 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.max_background_jobs: 2 2023-06-22T17:21:11.647 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.max_background_compactions: -1 2023-06-22T17:21:11.647 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.max_subcompactions: 1 2023-06-22T17:21:11.647 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-06-22T17:21:11.647 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-06-22T17:21:11.648 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.delayed_write_rate : 16777216 2023-06-22T17:21:11.648 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.max_total_wal_size: 0 2023-06-22T17:21:11.648 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-06-22T17:21:11.649 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.stats_dump_period_sec: 600 2023-06-22T17:21:11.649 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.stats_persist_period_sec: 600 2023-06-22T17:21:11.649 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-06-22T17:21:11.649 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.max_open_files: -1 2023-06-22T17:21:11.650 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.bytes_per_sync: 0 2023-06-22T17:21:11.650 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.wal_bytes_per_sync: 0 2023-06-22T17:21:11.650 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.strict_bytes_per_sync: 0 2023-06-22T17:21:11.650 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.compaction_readahead_size: 0 2023-06-22T17:21:11.651 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.max_background_flushes: -1 2023-06-22T17:21:11.651 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Compression algorithms supported: 2023-06-22T17:21:11.651 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-06-22T17:21:11.652 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: kZSTD supported: 0 2023-06-22T17:21:11.652 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: kXpressCompression supported: 0 2023-06-22T17:21:11.652 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: kLZ4HCCompression supported: 1 2023-06-22T17:21:11.652 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: kZlibCompression supported: 1 2023-06-22T17:21:11.653 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: kSnappyCompression supported: 1 2023-06-22T17:21:11.653 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: kLZ4Compression supported: 1 2023-06-22T17:21:11.653 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: kBZip2Compression supported: 0 2023-06-22T17:21:11.653 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Fast CRC32 supported: Supported on x86 2023-06-22T17:21:11.654 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 2023-06-22T17:21:11.654 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2023-06-22T17:21:11.654 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-06-22T17:21:11.655 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.merge_operator: 2023-06-22T17:21:11.655 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.compaction_filter: None 2023-06-22T17:21:11.655 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.compaction_filter_factory: None 2023-06-22T17:21:11.655 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.sst_partitioner_factory: None 2023-06-22T17:21:11.656 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.memtable_factory: SkipListFactory 2023-06-22T17:21:11.656 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.table_factory: BlockBasedTable 2023-06-22T17:21:11.656 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55eccbd04d30) 2023-06-22T17:21:11.656 INFO:journalctl@ceph.mon.b.smithi101.stdout: cache_index_and_filter_blocks: 1 2023-06-22T17:21:11.657 INFO:journalctl@ceph.mon.b.smithi101.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-06-22T17:21:11.657 INFO:journalctl@ceph.mon.b.smithi101.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-06-22T17:21:11.657 INFO:journalctl@ceph.mon.b.smithi101.stdout: pin_top_level_index_and_filter: 1 2023-06-22T17:21:11.657 INFO:journalctl@ceph.mon.b.smithi101.stdout: index_type: 0 2023-06-22T17:21:11.658 INFO:journalctl@ceph.mon.b.smithi101.stdout: data_block_index_type: 0 2023-06-22T17:21:11.658 INFO:journalctl@ceph.mon.b.smithi101.stdout: index_shortening: 1 2023-06-22T17:21:11.658 INFO:journalctl@ceph.mon.b.smithi101.stdout: data_block_hash_table_util_ratio: 0.750000 2023-06-22T17:21:11.658 INFO:journalctl@ceph.mon.b.smithi101.stdout: hash_index_allow_collision: 1 2023-06-22T17:21:11.658 INFO:journalctl@ceph.mon.b.smithi101.stdout: checksum: 1 2023-06-22T17:21:11.659 INFO:journalctl@ceph.mon.b.smithi101.stdout: no_block_cache: 0 2023-06-22T17:21:11.659 INFO:journalctl@ceph.mon.b.smithi101.stdout: block_cache: 0x55eccbdbd1f0 2023-06-22T17:21:11.659 INFO:journalctl@ceph.mon.b.smithi101.stdout: block_cache_name: BinnedLRUCache 2023-06-22T17:21:11.659 INFO:journalctl@ceph.mon.b.smithi101.stdout: block_cache_options: 2023-06-22T17:21:11.660 INFO:journalctl@ceph.mon.b.smithi101.stdout: capacity : 536870912 2023-06-22T17:21:11.660 INFO:journalctl@ceph.mon.b.smithi101.stdout: num_shard_bits : 4 2023-06-22T17:21:11.660 INFO:journalctl@ceph.mon.b.smithi101.stdout: strict_capacity_limit : 0 2023-06-22T17:21:11.660 INFO:journalctl@ceph.mon.b.smithi101.stdout: high_pri_pool_ratio: 0.000 2023-06-22T17:21:11.661 INFO:journalctl@ceph.mon.b.smithi101.stdout: block_cache_compressed: (nil) 2023-06-22T17:21:11.661 INFO:journalctl@ceph.mon.b.smithi101.stdout: persistent_cache: (nil) 2023-06-22T17:21:11.661 INFO:journalctl@ceph.mon.b.smithi101.stdout: block_size: 4096 2023-06-22T17:21:11.661 INFO:journalctl@ceph.mon.b.smithi101.stdout: block_size_deviation: 10 2023-06-22T17:21:11.662 INFO:journalctl@ceph.mon.b.smithi101.stdout: block_restart_interval: 16 2023-06-22T17:21:11.662 INFO:journalctl@ceph.mon.b.smithi101.stdout: index_block_restart_interval: 1 2023-06-22T17:21:11.662 INFO:journalctl@ceph.mon.b.smithi101.stdout: metadata_block_size: 4096 2023-06-22T17:21:11.663 INFO:journalctl@ceph.mon.b.smithi101.stdout: partition_filters: 0 2023-06-22T17:21:11.663 INFO:journalctl@ceph.mon.b.smithi101.stdout: use_delta_encoding: 1 2023-06-22T17:21:11.663 INFO:journalctl@ceph.mon.b.smithi101.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2023-06-22T17:21:11.663 INFO:journalctl@ceph.mon.b.smithi101.stdout: whole_key_filtering: 1 2023-06-22T17:21:11.664 INFO:journalctl@ceph.mon.b.smithi101.stdout: verify_compression: 0 2023-06-22T17:21:11.664 INFO:journalctl@ceph.mon.b.smithi101.stdout: read_amp_bytes_per_bit: 0 2023-06-22T17:21:11.664 INFO:journalctl@ceph.mon.b.smithi101.stdout: format_version: 4 2023-06-22T17:21:11.664 INFO:journalctl@ceph.mon.b.smithi101.stdout: enable_index_compression: 1 2023-06-22T17:21:11.664 INFO:journalctl@ceph.mon.b.smithi101.stdout: block_align: 0 2023-06-22T17:21:11.665 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.write_buffer_size: 33554432 2023-06-22T17:21:11.665 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.max_write_buffer_number: 2 2023-06-22T17:21:11.665 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.compression: NoCompression 2023-06-22T17:21:11.665 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.bottommost_compression: Disabled 2023-06-22T17:21:11.665 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.prefix_extractor: nullptr 2023-06-22T17:21:11.665 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-06-22T17:21:11.665 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.num_levels: 7 2023-06-22T17:21:11.665 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-06-22T17:21:11.666 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-06-22T17:21:11.666 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-06-22T17:21:11.666 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-06-22T17:21:11.666 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-06-22T17:21:11.666 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-06-22T17:21:11.666 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-06-22T17:21:11.666 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-06-22T17:21:11.667 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-06-22T17:21:11.667 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-06-22T17:21:11.667 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.compression_opts.window_bits: -14 2023-06-22T17:21:11.667 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.compression_opts.level: 32767 2023-06-22T17:21:11.667 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.compression_opts.strategy: 0 2023-06-22T17:21:11.668 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-06-22T17:21:11.669 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-06-22T17:21:11.669 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-06-22T17:21:11.669 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.compression_opts.enabled: false 2023-06-22T17:21:11.669 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-06-22T17:21:11.669 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-06-22T17:21:11.670 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-06-22T17:21:11.670 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.target_file_size_base: 67108864 2023-06-22T17:21:11.670 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.target_file_size_multiplier: 1 2023-06-22T17:21:11.670 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-06-22T17:21:11.670 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-06-22T17:21:11.670 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-06-22T17:21:11.671 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-06-22T17:21:11.671 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-06-22T17:21:11.671 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-06-22T17:21:11.672 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-06-22T17:21:11.672 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-06-22T17:21:11.672 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-06-22T17:21:11.672 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-06-22T17:21:11.672 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-06-22T17:21:11.672 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-06-22T17:21:11.672 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.arena_block_size: 4194304 2023-06-22T17:21:11.673 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-06-22T17:21:11.673 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-06-22T17:21:11.673 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-06-22T17:21:11.673 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.disable_auto_compactions: 0 2023-06-22T17:21:11.673 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-06-22T17:21:11.674 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-06-22T17:21:11.674 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-06-22T17:21:11.674 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-06-22T17:21:11.674 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-06-22T17:21:11.674 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-06-22T17:21:11.674 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-06-22T17:21:11.674 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-06-22T17:21:11.675 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-06-22T17:21:11.675 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-06-22T17:21:11.675 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.table_properties_collectors: 2023-06-22T17:21:11.675 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.inplace_update_support: 0 2023-06-22T17:21:11.676 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.inplace_update_num_locks: 10000 2023-06-22T17:21:11.676 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-06-22T17:21:11.676 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-06-22T17:21:11.676 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.memtable_huge_page_size: 0 2023-06-22T17:21:11.676 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.bloom_locality: 0 2023-06-22T17:21:11.676 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.max_successive_merges: 0 2023-06-22T17:21:11.677 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.optimize_filters_for_hits: 0 2023-06-22T17:21:11.677 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.paranoid_file_checks: 0 2023-06-22T17:21:11.677 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.force_consistency_checks: 1 2023-06-22T17:21:11.677 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.report_bg_io_stats: 0 2023-06-22T17:21:11.677 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.ttl: 2592000 2023-06-22T17:21:11.677 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.periodic_compaction_seconds: 0 2023-06-22T17:21:11.678 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.enable_blob_files: false 2023-06-22T17:21:11.678 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.min_blob_size: 0 2023-06-22T17:21:11.678 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.blob_file_size: 268435456 2023-06-22T17:21:11.678 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.blob_compression_type: NoCompression 2023-06-22T17:21:11.678 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.enable_blob_garbage_collection: false 2023-06-22T17:21:11.678 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-06-22T17:21:11.678 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-06-22T17:21:11.678 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2023-06-22T17:21:11.679 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2023-06-22T17:21:11.679 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: EVENT_LOG_v1 {"time_micros": 1687454471159839, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-06-22T17:21:11.679 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2023-06-22T17:21:11.679 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-06-22T17:21:11.679 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: EVENT_LOG_v1 {"time_micros": 1687454471160105, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1670, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1687454471, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "68edcabe-205c-4e65-829f-ac6030cb59c7", "db_session_id": "V5JYHCUM82SCDRYU8BLK"}} 2023-06-22T17:21:11.679 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2023-06-22T17:21:11.679 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: EVENT_LOG_v1 {"time_micros": 1687454471160456, "job": 1, "event": "recovery_finished"} 2023-06-22T17:21:11.680 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-06-22T17:21:11.680 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55eccbd51180 2023-06-22T17:21:11.680 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: DB pointer 0x55eccbd6c000 2023-06-22T17:21:11.680 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2023-06-22T17:21:11.680 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: rocksdb: [db/db_impl/db_impl.cc:903] 2023-06-22T17:21:11.680 INFO:journalctl@ceph.mon.b.smithi101.stdout: ** DB Stats ** 2023-06-22T17:21:11.680 INFO:journalctl@ceph.mon.b.smithi101.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-06-22T17:21:11.680 INFO:journalctl@ceph.mon.b.smithi101.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-06-22T17:21:11.681 INFO:journalctl@ceph.mon.b.smithi101.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-06-22T17:21:11.681 INFO:journalctl@ceph.mon.b.smithi101.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-06-22T17:21:11.681 INFO:journalctl@ceph.mon.b.smithi101.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-06-22T17:21:11.681 INFO:journalctl@ceph.mon.b.smithi101.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-06-22T17:21:11.681 INFO:journalctl@ceph.mon.b.smithi101.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-06-22T17:21:11.681 INFO:journalctl@ceph.mon.b.smithi101.stdout: 2023-06-22T17:21:11.681 INFO:journalctl@ceph.mon.b.smithi101.stdout: ** Compaction Stats [default] ** 2023-06-22T17:21:11.681 INFO:journalctl@ceph.mon.b.smithi101.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-06-22T17:21:11.682 INFO:journalctl@ceph.mon.b.smithi101.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-06-22T17:21:11.682 INFO:journalctl@ceph.mon.b.smithi101.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 6.4 0.00 0.00 1 0.000 0 0 2023-06-22T17:21:11.682 INFO:journalctl@ceph.mon.b.smithi101.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 6.4 0.00 0.00 1 0.000 0 0 2023-06-22T17:21:11.682 INFO:journalctl@ceph.mon.b.smithi101.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 6.4 0.00 0.00 1 0.000 0 0 2023-06-22T17:21:11.682 INFO:journalctl@ceph.mon.b.smithi101.stdout: 2023-06-22T17:21:11.682 INFO:journalctl@ceph.mon.b.smithi101.stdout: ** Compaction Stats [default] ** 2023-06-22T17:21:11.682 INFO:journalctl@ceph.mon.b.smithi101.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-06-22T17:21:11.683 INFO:journalctl@ceph.mon.b.smithi101.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-06-22T17:21:11.683 INFO:journalctl@ceph.mon.b.smithi101.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 6.4 0.00 0.00 1 0.000 0 0 2023-06-22T17:21:11.683 INFO:journalctl@ceph.mon.b.smithi101.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-06-22T17:21:11.683 INFO:journalctl@ceph.mon.b.smithi101.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-06-22T17:21:11.683 INFO:journalctl@ceph.mon.b.smithi101.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-06-22T17:21:11.683 INFO:journalctl@ceph.mon.b.smithi101.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-06-22T17:21:11.683 INFO:journalctl@ceph.mon.b.smithi101.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-06-22T17:21:11.684 INFO:journalctl@ceph.mon.b.smithi101.stdout: AddFile(Keys): cumulative 0, interval 0 2023-06-22T17:21:11.684 INFO:journalctl@ceph.mon.b.smithi101.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-06-22T17:21:11.684 INFO:journalctl@ceph.mon.b.smithi101.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-06-22T17:21:11.684 INFO:journalctl@ceph.mon.b.smithi101.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-06-22T17:21:11.684 INFO:journalctl@ceph.mon.b.smithi101.stdout: 2023-06-22T17:21:11.684 INFO:journalctl@ceph.mon.b.smithi101.stdout: ** File Read Latency Histogram By Level [default] ** 2023-06-22T17:21:11.685 INFO:journalctl@ceph.mon.b.smithi101.stdout: 2023-06-22T17:21:11.685 INFO:journalctl@ceph.mon.b.smithi101.stdout: ** Compaction Stats [default] ** 2023-06-22T17:21:11.685 INFO:journalctl@ceph.mon.b.smithi101.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-06-22T17:21:11.685 INFO:journalctl@ceph.mon.b.smithi101.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-06-22T17:21:11.685 INFO:journalctl@ceph.mon.b.smithi101.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 6.4 0.00 0.00 1 0.000 0 0 2023-06-22T17:21:11.685 INFO:journalctl@ceph.mon.b.smithi101.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 6.4 0.00 0.00 1 0.000 0 0 2023-06-22T17:21:11.686 INFO:journalctl@ceph.mon.b.smithi101.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-06-22T17:21:11.686 INFO:journalctl@ceph.mon.b.smithi101.stdout: 2023-06-22T17:21:11.686 INFO:journalctl@ceph.mon.b.smithi101.stdout: ** Compaction Stats [default] ** 2023-06-22T17:21:11.686 INFO:journalctl@ceph.mon.b.smithi101.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-06-22T17:21:11.686 INFO:journalctl@ceph.mon.b.smithi101.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-06-22T17:21:11.686 INFO:journalctl@ceph.mon.b.smithi101.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 6.4 0.00 0.00 1 0.000 0 0 2023-06-22T17:21:11.686 INFO:journalctl@ceph.mon.b.smithi101.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-06-22T17:21:11.686 INFO:journalctl@ceph.mon.b.smithi101.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-06-22T17:21:11.687 INFO:journalctl@ceph.mon.b.smithi101.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-06-22T17:21:11.687 INFO:journalctl@ceph.mon.b.smithi101.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-06-22T17:21:11.687 INFO:journalctl@ceph.mon.b.smithi101.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-06-22T17:21:11.687 INFO:journalctl@ceph.mon.b.smithi101.stdout: AddFile(Keys): cumulative 0, interval 0 2023-06-22T17:21:11.687 INFO:journalctl@ceph.mon.b.smithi101.stdout: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-06-22T17:21:11.687 INFO:journalctl@ceph.mon.b.smithi101.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-06-22T17:21:11.688 INFO:journalctl@ceph.mon.b.smithi101.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-06-22T17:21:11.688 INFO:journalctl@ceph.mon.b.smithi101.stdout: 2023-06-22T17:21:11.688 INFO:journalctl@ceph.mon.b.smithi101.stdout: ** File Read Latency Histogram By Level [default] ** 2023-06-22T17:21:11.688 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: mon.b does not exist in monmap, will attempt to join an existing cluster 2023-06-22T17:21:11.688 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: using public_addr v2:172.21.15.101:0/0 -> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] 2023-06-22T17:21:11.688 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: starting mon.b rank -1 at public addrs [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] at bind addrs [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid d88002e2-1120-11ee-9b2b-001a4aab830c 2023-06-22T17:21:11.689 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: mon.b@-1(???) e0 preinit fsid d88002e2-1120-11ee-9b2b-001a4aab830c 2023-06-22T17:21:11.689 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 bash[85443]: c51fb97fabf52865327516731efc0128db942b36ebf8c8171409a95642ca333f 2023-06-22T17:21:11.689 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: mon.b@-1(synchronizing).mds e1 new map 2023-06-22T17:21:11.689 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: mon.b@-1(synchronizing).mds e1 print_map 2023-06-22T17:21:11.689 INFO:journalctl@ceph.mon.b.smithi101.stdout: e1 2023-06-22T17:21:11.689 INFO:journalctl@ceph.mon.b.smithi101.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-06-22T17:21:11.689 INFO:journalctl@ceph.mon.b.smithi101.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-06-22T17:21:11.689 INFO:journalctl@ceph.mon.b.smithi101.stdout: legacy client fscid: -1 2023-06-22T17:21:11.690 INFO:journalctl@ceph.mon.b.smithi101.stdout: 2023-06-22T17:21:11.690 INFO:journalctl@ceph.mon.b.smithi101.stdout: No filesystems configured 2023-06-22T17:21:11.690 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-06-22T17:21:11.690 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-06-22T17:21:11.690 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-06-22T17:21:11.690 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-06-22T17:21:11.690 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-06-22T17:21:11.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-06-22T17:21:11.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-06-22T17:21:11.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-06-22T17:21:11.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-06-22T17:21:11.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-06-22T17:21:11.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: Updating smithi101:/var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/config/ceph.conf 2023-06-22T17:21:11.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: Updating smithi101:/etc/ceph/ceph.client.admin.keyring 2023-06-22T17:21:11.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: Deploying cephadm binary to smithi133 2023-06-22T17:21:11.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: Updating smithi101:/var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/config/ceph.client.admin.keyring 2023-06-22T17:21:11.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:11.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:11.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:21:11.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: Added host smithi133 2023-06-22T17:21:11.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:11.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:11.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:11.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-06-22T17:21:11.694 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:11.694 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:11.694 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/2576570957' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-06-22T17:21:11.694 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/2576570957' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-06-22T17:21:11.694 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: osdmap e4: 0 total, 0 up, 0 in 2023-06-22T17:21:11.694 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:11.695 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:11.695 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:11.695 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:11.695 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi133", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:21:11.695 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:11.695 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:21:11.695 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: Updating smithi133:/etc/ceph/ceph.conf 2023-06-22T17:21:11.696 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:11.696 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: Updating smithi133:/var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/config/ceph.conf 2023-06-22T17:21:11.696 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:11.696 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:11.696 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:11.696 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: Updating smithi133:/etc/ceph/ceph.client.admin.keyring 2023-06-22T17:21:11.697 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: Updating smithi133:/var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/config/ceph.client.admin.keyring 2023-06-22T17:21:11.697 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:11.697 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:11.697 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:21:11.697 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:11.697 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:21:11.697 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:11.698 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-06-22T17:21:11.698 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:11.698 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi039:172.21.15.39=a;smithi101:172.21.15.101=b;smithi133:172.21.15.133=c", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:21:11.698 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: Saving service mon spec with placement smithi039:172.21.15.39=a;smithi101:172.21.15.101=b;smithi133:172.21.15.133=c;count:3 2023-06-22T17:21:11.698 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: Deploying daemon mon.c on smithi133 2023-06-22T17:21:11.698 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:11.699 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-22T17:21:11.699 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: mon.a calling monitor election 2023-06-22T17:21:11.699 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-22T17:21:11.699 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:11.699 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-22T17:21:11.699 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-22T17:21:11.699 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: mon.c calling monitor election 2023-06-22T17:21:11.700 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:11.700 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-22T17:21:11.700 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-22T17:21:11.700 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:11.700 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-22T17:21:11.700 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-06-22T17:21:11.700 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: monmap e2: 2 mons at {a=[v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0],c=[v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0]} removed_ranks: {} 2023-06-22T17:21:11.701 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: fsmap 2023-06-22T17:21:11.701 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: osdmap e4: 0 total, 0 up, 0 in 2023-06-22T17:21:11.701 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: mgrmap e13: x(active, since 40s) 2023-06-22T17:21:11.701 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: overall HEALTH_OK 2023-06-22T17:21:11.701 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:11.701 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:11.701 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:11.701 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-06-22T17:21:11.702 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:11.702 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: Deploying daemon mon.b on smithi101 2023-06-22T17:21:11.702 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-22T17:21:11.702 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:11.702 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: from='client.? 172.21.15.133:0/1022419216' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-06-22T17:21:11.702 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-06-22T17:21:11.702 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 ceph-mon[85507]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2023-06-22T17:21:11.703 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:11 smithi101 systemd[1]: Started Ceph mon.b for d88002e2-1120-11ee-9b2b-001a4aab830c. 2023-06-22T17:21:16.269 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.268+0000 7f2e89eac700 1 -- 172.21.15.133:0/3355246153 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e70003580 msgr2=0x7f2e70005a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:21:16.270 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.268+0000 7f2e89eac700 1 --2- 172.21.15.133:0/3355246153 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e70003580 0x7f2e70005a00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2e840fb5c0 tx=0x7f2e7400b040 comp rx=0 tx=0).stop 2023-06-22T17:21:16.270 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.268+0000 7f2e89eac700 1 -- 172.21.15.133:0/3355246153 shutdown_connections 2023-06-22T17:21:16.270 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.268+0000 7f2e89eac700 1 --2- 172.21.15.133:0/3355246153 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e70003580 0x7f2e70005a00 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:16.270 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.268+0000 7f2e89eac700 1 -- 172.21.15.133:0/3355246153 >> 172.21.15.133:0/3355246153 conn(0x7f2e840f7250 msgr2=0x7f2e840f9660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:21:16.270 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.268+0000 7f2e89eac700 1 -- 172.21.15.133:0/3355246153 shutdown_connections 2023-06-22T17:21:16.270 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.268+0000 7f2e89eac700 1 -- 172.21.15.133:0/3355246153 wait complete. 2023-06-22T17:21:16.272 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.269+0000 7f2e89eac700 1 Processor -- start 2023-06-22T17:21:16.272 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.269+0000 7f2e89eac700 1 -- start start 2023-06-22T17:21:16.272 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.269+0000 7f2e89eac700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e70003580 0x7f2e84112ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:21:16.272 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.269+0000 7f2e89eac700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2e84113020 0x7f2e8410c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:21:16.272 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.269+0000 7f2e89eac700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2e8410cf30 0x7f2e8410d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:21:16.272 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.269+0000 7f2e89eac700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e84113670 con 0x7f2e70003580 2023-06-22T17:21:16.273 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.269+0000 7f2e89eac700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e841198e0 con 0x7f2e8410cf30 2023-06-22T17:21:16.273 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.269+0000 7f2e89eac700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e84119a20 con 0x7f2e84113020 2023-06-22T17:21:16.273 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.269+0000 7f2e837fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e70003580 0x7f2e84112ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:21:16.273 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.269+0000 7f2e7bfff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2e84113020 0x7f2e8410c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:21:16.273 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.269+0000 7f2e837fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e70003580 0x7f2e84112ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.133:33470/0 (socket says 172.21.15.133:33470) 2023-06-22T17:21:16.273 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.269+0000 7f2e837fe700 1 -- 172.21.15.133:0/985242877 learned_addr learned my addr 172.21.15.133:0/985242877 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-06-22T17:21:16.273 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.269+0000 7f2e7bfff700 1 -- 172.21.15.133:0/985242877 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2e84113020 msgr2=0x7f2e8410c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-06-22T17:21:16.274 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.270+0000 7f2e7bfff700 1 -- 172.21.15.133:0/985242877 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2e84113020 msgr2=0x7f2e8410c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-06-22T17:21:16.274 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.269+0000 7f2e83fff700 1 --2- 172.21.15.133:0/985242877 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2e8410cf30 0x7f2e8410d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:21:16.274 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.270+0000 7f2e7bfff700 1 --2- 172.21.15.133:0/985242877 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2e84113020 0x7f2e8410c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-06-22T17:21:16.274 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.270+0000 7f2e7bfff700 1 --2- 172.21.15.133:0/985242877 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2e84113020 0x7f2e8410c9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T17:21:16.274 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.270+0000 7f2e83fff700 1 -- 172.21.15.133:0/985242877 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2e8410cf30 msgr2=0x7f2e8410d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2023-06-22T17:21:16.274 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.270+0000 7f2e83fff700 1 -- 172.21.15.133:0/985242877 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2e8410cf30 msgr2=0x7f2e8410d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-06-22T17:21:16.274 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.270+0000 7f2e83fff700 1 --2- 172.21.15.133:0/985242877 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2e8410cf30 0x7f2e8410d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-06-22T17:21:16.275 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.270+0000 7f2e83fff700 1 --2- 172.21.15.133:0/985242877 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2e8410cf30 0x7f2e8410d3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T17:21:16.275 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.270+0000 7f2e837fe700 1 -- 172.21.15.133:0/985242877 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2e84113020 msgr2=0x7f2e8410c9f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-06-22T17:21:16.275 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.270+0000 7f2e837fe700 1 --2- 172.21.15.133:0/985242877 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2e84113020 0x7f2e8410c9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:16.275 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.270+0000 7f2e837fe700 1 -- 172.21.15.133:0/985242877 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2e8410cf30 msgr2=0x7f2e8410d3d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-06-22T17:21:16.275 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.270+0000 7f2e837fe700 1 --2- 172.21.15.133:0/985242877 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2e8410cf30 0x7f2e8410d3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:16.275 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.270+0000 7f2e837fe700 1 -- 172.21.15.133:0/985242877 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e74005040 con 0x7f2e70003580 2023-06-22T17:21:16.275 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.270+0000 7f2e837fe700 1 --2- 172.21.15.133:0/985242877 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e70003580 0x7f2e84112ae0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2e74000ea0 tx=0x7f2e7400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:21:16.276 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.274+0000 7f2e80ff9700 1 -- 172.21.15.133:0/985242877 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e740098c0 con 0x7f2e70003580 2023-06-22T17:21:16.277 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.274+0000 7f2e89eac700 1 -- 172.21.15.133:0/985242877 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e84119ca0 con 0x7f2e70003580 2023-06-22T17:21:16.277 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.274+0000 7f2e89eac700 1 -- 172.21.15.133:0/985242877 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e8411a190 con 0x7f2e70003580 2023-06-22T17:21:16.277 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.274+0000 7f2e80ff9700 1 -- 172.21.15.133:0/985242877 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f2e74009a20 con 0x7f2e70003580 2023-06-22T17:21:16.278 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.274+0000 7f2e80ff9700 1 -- 172.21.15.133:0/985242877 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e74021430 con 0x7f2e70003580 2023-06-22T17:21:16.279 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.275+0000 7f2e80ff9700 1 -- 172.21.15.133:0/985242877 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f2e74021650 con 0x7f2e70003580 2023-06-22T17:21:16.279 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.275+0000 7f2e80ff9700 1 --2- 172.21.15.133:0/985242877 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f2e64036b30 0x7f2e64038fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:21:16.279 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.275+0000 7f2e80ff9700 1 -- 172.21.15.133:0/985242877 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f2e7404bbc0 con 0x7f2e70003580 2023-06-22T17:21:16.279 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.275+0000 7f2e7bfff700 1 --2- 172.21.15.133:0/985242877 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f2e64036b30 0x7f2e64038fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:21:16.279 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.275+0000 7f2e89eac700 1 -- 172.21.15.133:0/985242877 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e70002130 con 0x7f2e70003580 2023-06-22T17:21:16.280 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.276+0000 7f2e7bfff700 1 --2- 172.21.15.133:0/985242877 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f2e64036b30 0x7f2e64038fe0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c006f30 tx=0x7f2e6c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:21:16.282 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.281+0000 7f2e80ff9700 1 -- 172.21.15.133:0/985242877 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f2e740323a0 con 0x7f2e70003580 2023-06-22T17:21:16.555 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.554+0000 7f2e89eac700 1 -- 172.21.15.133:0/985242877 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2e700033f0 con 0x7f2e70003580 2023-06-22T17:21:16.556 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.555+0000 7f2e80ff9700 1 -- 172.21.15.133:0/985242877 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1269 (secure 0 0 0) 0x7f2e74049030 con 0x7f2e70003580 2023-06-22T17:21:16.556 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-06-22T17:21:16.557 INFO:teuthology.orchestra.run.smithi133.stdout:{"epoch":3,"fsid":"d88002e2-1120-11ee-9b2b-001a4aab830c","modified":"2023-06-22T17:21:11.181383Z","created":"2023-06-22T17:19:34.058140Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:3300","nonce":0},{"type":"v1","addr":"172.21.15.101:6789","nonce":0}]},"addr":"172.21.15.101:6789/0","public_addr":"172.21.15.101:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-06-22T17:21:16.560 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.558+0000 7f2e89eac700 1 -- 172.21.15.133:0/985242877 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f2e64036b30 msgr2=0x7f2e64038fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:21:16.560 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.558+0000 7f2e89eac700 1 --2- 172.21.15.133:0/985242877 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f2e64036b30 0x7f2e64038fe0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c006f30 tx=0x7f2e6c008040 comp rx=0 tx=0).stop 2023-06-22T17:21:16.560 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.558+0000 7f2e89eac700 1 -- 172.21.15.133:0/985242877 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e70003580 msgr2=0x7f2e84112ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:21:16.560 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.558+0000 7f2e89eac700 1 --2- 172.21.15.133:0/985242877 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e70003580 0x7f2e84112ae0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2e74000ea0 tx=0x7f2e7400b040 comp rx=0 tx=0).stop 2023-06-22T17:21:16.560 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.558+0000 7f2e89eac700 1 -- 172.21.15.133:0/985242877 shutdown_connections 2023-06-22T17:21:16.560 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.558+0000 7f2e89eac700 1 --2- 172.21.15.133:0/985242877 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f2e64036b30 0x7f2e64038fe0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:16.561 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.558+0000 7f2e89eac700 1 --2- 172.21.15.133:0/985242877 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2e8410cf30 0x7f2e8410d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:16.561 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.558+0000 7f2e89eac700 1 --2- 172.21.15.133:0/985242877 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e70003580 0x7f2e84112ae0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:16.561 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.558+0000 7f2e89eac700 1 --2- 172.21.15.133:0/985242877 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2e84113020 0x7f2e8410c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:16.561 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.558+0000 7f2e89eac700 1 -- 172.21.15.133:0/985242877 >> 172.21.15.133:0/985242877 conn(0x7f2e840f7250 msgr2=0x7f2e840ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:21:16.561 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.559+0000 7f2e89eac700 1 -- 172.21.15.133:0/985242877 shutdown_connections 2023-06-22T17:21:16.561 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:16.559+0000 7f2e89eac700 1 -- 172.21.15.133:0/985242877 wait complete. 2023-06-22T17:21:16.562 INFO:teuthology.orchestra.run.smithi133.stderr:dumped monmap epoch 3 2023-06-22T17:21:16.589 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:16 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-22T17:21:16.590 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:16 smithi133 ceph-mon[86268]: mon.c calling monitor election 2023-06-22T17:21:16.590 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:16 smithi133 ceph-mon[86268]: mon.a calling monitor election 2023-06-22T17:21:16.590 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:16 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-22T17:21:16.591 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:16 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-22T17:21:16.591 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:16 smithi133 ceph-mon[86268]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:16.591 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:16 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-22T17:21:16.591 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:16 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-22T17:21:16.591 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:16 smithi133 ceph-mon[86268]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:16.591 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:16 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-22T17:21:16.592 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:16 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-22T17:21:16.592 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:16 smithi133 ceph-mon[86268]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:16.592 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:16 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-22T17:21:16.592 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:16 smithi133 ceph-mon[86268]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-06-22T17:21:16.592 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:16 smithi133 ceph-mon[86268]: monmap e3: 3 mons at {a=[v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0],b=[v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0],c=[v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0]} removed_ranks: {} 2023-06-22T17:21:16.592 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:16 smithi133 ceph-mon[86268]: fsmap 2023-06-22T17:21:16.592 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:16 smithi133 ceph-mon[86268]: osdmap e4: 0 total, 0 up, 0 in 2023-06-22T17:21:16.592 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:16 smithi133 ceph-mon[86268]: mgrmap e13: x(active, since 48s) 2023-06-22T17:21:16.593 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:16 smithi133 ceph-mon[86268]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-06-22T17:21:16.593 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:16 smithi133 ceph-mon[86268]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-06-22T17:21:16.593 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:16 smithi133 ceph-mon[86268]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-06-22T17:21:16.593 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:16 smithi133 ceph-mon[86268]: mon.b (rank 2) addr [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] is down (out of quorum) 2023-06-22T17:21:16.593 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:16 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:16.593 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:16 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:16.593 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:16 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:16.594 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:16 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:16.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:16 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-22T17:21:16.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:16 smithi039 ceph-mon[84805]: mon.c calling monitor election 2023-06-22T17:21:16.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:16 smithi039 ceph-mon[84805]: mon.a calling monitor election 2023-06-22T17:21:16.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:16 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-22T17:21:16.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:16 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-22T17:21:16.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:16 smithi039 ceph-mon[84805]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:16.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:16 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-22T17:21:16.685 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:16 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-22T17:21:16.685 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:16 smithi039 ceph-mon[84805]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:16.685 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:16 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-22T17:21:16.685 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:16 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-22T17:21:16.685 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:16 smithi039 ceph-mon[84805]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:16.685 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:16 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-22T17:21:16.686 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:16 smithi039 ceph-mon[84805]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-06-22T17:21:16.686 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:16 smithi039 ceph-mon[84805]: monmap e3: 3 mons at {a=[v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0],b=[v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0],c=[v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0]} removed_ranks: {} 2023-06-22T17:21:16.686 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:16 smithi039 ceph-mon[84805]: fsmap 2023-06-22T17:21:16.686 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:16 smithi039 ceph-mon[84805]: osdmap e4: 0 total, 0 up, 0 in 2023-06-22T17:21:16.686 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:16 smithi039 ceph-mon[84805]: mgrmap e13: x(active, since 48s) 2023-06-22T17:21:16.686 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:16 smithi039 ceph-mon[84805]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-06-22T17:21:16.686 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:16 smithi039 ceph-mon[84805]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-06-22T17:21:16.687 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:16 smithi039 ceph-mon[84805]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-06-22T17:21:16.687 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:16 smithi039 ceph-mon[84805]: mon.b (rank 2) addr [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] is down (out of quorum) 2023-06-22T17:21:16.687 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:16 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:16.687 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:16 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:16.687 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:16 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:16.687 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:16 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:16.951 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-06-22T17:21:16.951 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph config generate-minimal-conf 2023-06-22T17:21:17.560 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-22T17:21:17.669 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:17 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:21:17.669 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:17 smithi133 ceph-mon[86268]: from='client.? 172.21.15.133:0/985242877' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-06-22T17:21:17.669 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:17 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-22T17:21:17.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:17 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:21:17.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:17 smithi039 ceph-mon[84805]: from='client.? 172.21.15.133:0/985242877' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-06-22T17:21:17.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:17 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-22T17:21:18.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:18 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-22T17:21:18.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:18 smithi101 ceph-mon[85507]: mon.c calling monitor election 2023-06-22T17:21:18.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:18 smithi101 ceph-mon[85507]: mon.a calling monitor election 2023-06-22T17:21:18.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:18 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-22T17:21:18.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:18 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-22T17:21:18.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:18 smithi101 ceph-mon[85507]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:18.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:18 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-22T17:21:18.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:18 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-22T17:21:18.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:18 smithi101 ceph-mon[85507]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:18.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:18 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-22T17:21:18.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:18 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-22T17:21:18.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:18 smithi101 ceph-mon[85507]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:18.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:18 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-22T17:21:18.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:18 smithi101 ceph-mon[85507]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-06-22T17:21:18.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:18 smithi101 ceph-mon[85507]: monmap e3: 3 mons at {a=[v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0],b=[v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0],c=[v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0]} removed_ranks: {} 2023-06-22T17:21:18.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:18 smithi101 ceph-mon[85507]: fsmap 2023-06-22T17:21:18.628 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:18 smithi101 ceph-mon[85507]: osdmap e4: 0 total, 0 up, 0 in 2023-06-22T17:21:18.628 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:18 smithi101 ceph-mon[85507]: mgrmap e13: x(active, since 48s) 2023-06-22T17:21:18.628 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:18 smithi101 ceph-mon[85507]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-06-22T17:21:18.628 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:18 smithi101 ceph-mon[85507]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-06-22T17:21:18.628 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:18 smithi101 ceph-mon[85507]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-06-22T17:21:18.628 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:18 smithi101 ceph-mon[85507]: mon.b (rank 2) addr [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] is down (out of quorum) 2023-06-22T17:21:18.628 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:18 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:18.629 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:18 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:18.629 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:18 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:18.629 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:18 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:18.629 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:18 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:21:18.629 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:18 smithi101 ceph-mon[85507]: from='client.? 172.21.15.133:0/985242877' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-06-22T17:21:18.629 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:18 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-22T17:21:18.862 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.858+0000 7f37cdd9d700 1 -- 172.21.15.39:0/2010828339 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37c81026c0 msgr2=0x7f37c8102a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:21:18.863 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.858+0000 7f37cdd9d700 1 --2- 172.21.15.39:0/2010828339 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37c81026c0 0x7f37c8102a90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f37b80048c0 tx=0x7f37b8009e30 comp rx=0 tx=0).stop 2023-06-22T17:21:18.863 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.858+0000 7f37cdd9d700 1 -- 172.21.15.39:0/2010828339 shutdown_connections 2023-06-22T17:21:18.863 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.858+0000 7f37cdd9d700 1 --2- 172.21.15.39:0/2010828339 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37c81026c0 0x7f37c8102a90 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:18.864 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.858+0000 7f37cdd9d700 1 -- 172.21.15.39:0/2010828339 >> 172.21.15.39:0/2010828339 conn(0x7f37c80f81e0 msgr2=0x7f37c80fa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:21:18.864 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.858+0000 7f37cdd9d700 1 -- 172.21.15.39:0/2010828339 shutdown_connections 2023-06-22T17:21:18.864 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.858+0000 7f37cdd9d700 1 -- 172.21.15.39:0/2010828339 wait complete. 2023-06-22T17:21:18.864 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.859+0000 7f37cdd9d700 1 Processor -- start 2023-06-22T17:21:18.865 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.859+0000 7f37cdd9d700 1 -- start start 2023-06-22T17:21:18.865 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.859+0000 7f37cdd9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37c81026c0 0x7f37c8112ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:21:18.865 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.859+0000 7f37cdd9d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f37c8113220 0x7f37c810cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:21:18.865 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.859+0000 7f37cdd9d700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f37c810d0c0 0x7f37c810d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:21:18.866 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.859+0000 7f37cdd9d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c8113880 con 0x7f37c81026c0 2023-06-22T17:21:18.866 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.859+0000 7f37cdd9d700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c8119b10 con 0x7f37c810d0c0 2023-06-22T17:21:18.866 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.859+0000 7f37cdd9d700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c8119c50 con 0x7f37c8113220 2023-06-22T17:21:18.866 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.859+0000 7f37c77fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37c81026c0 0x7f37c8112ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:21:18.866 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.859+0000 7f37c77fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37c81026c0 0x7f37c8112ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44294/0 (socket says 172.21.15.39:44294) 2023-06-22T17:21:18.867 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.859+0000 7f37c77fe700 1 -- 172.21.15.39:0/1411626813 learned_addr learned my addr 172.21.15.39:0/1411626813 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:21:18.867 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.859+0000 7f37c7fff700 1 --2- 172.21.15.39:0/1411626813 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f37c810d0c0 0x7f37c810d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:21:18.867 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.859+0000 7f37c6ffd700 1 --2- 172.21.15.39:0/1411626813 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f37c8113220 0x7f37c810cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:21:18.867 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.860+0000 7f37c77fe700 1 -- 172.21.15.39:0/1411626813 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f37c8113220 msgr2=0x7f37c810cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:21:18.867 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.860+0000 7f37c77fe700 1 --2- 172.21.15.39:0/1411626813 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f37c8113220 0x7f37c810cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:18.868 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.860+0000 7f37c77fe700 1 -- 172.21.15.39:0/1411626813 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f37c810d0c0 msgr2=0x7f37c810d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:21:18.868 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.860+0000 7f37c77fe700 1 --2- 172.21.15.39:0/1411626813 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f37c810d0c0 0x7f37c810d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:18.868 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.860+0000 7f37c77fe700 1 -- 172.21.15.39:0/1411626813 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b8005040 con 0x7f37c81026c0 2023-06-22T17:21:18.868 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.860+0000 7f37c6ffd700 1 --2- 172.21.15.39:0/1411626813 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f37c8113220 0x7f37c810cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T17:21:18.868 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.860+0000 7f37c7fff700 1 --2- 172.21.15.39:0/1411626813 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f37c810d0c0 0x7f37c810d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T17:21:18.869 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.860+0000 7f37c77fe700 1 --2- 172.21.15.39:0/1411626813 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37c81026c0 0x7f37c8112ce0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f37b8008fe0 tx=0x7f37b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:21:18.869 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.860+0000 7f37bffff700 1 -- 172.21.15.39:0/1411626813 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b800e040 con 0x7f37c81026c0 2023-06-22T17:21:18.869 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.860+0000 7f37bffff700 1 -- 172.21.15.39:0/1411626813 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f37b8009870 con 0x7f37c81026c0 2023-06-22T17:21:18.870 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.860+0000 7f37cdd9d700 1 -- 172.21.15.39:0/1411626813 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37c8119ea0 con 0x7f37c81026c0 2023-06-22T17:21:18.870 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.860+0000 7f37bffff700 1 -- 172.21.15.39:0/1411626813 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b8013e90 con 0x7f37c81026c0 2023-06-22T17:21:18.870 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.860+0000 7f37cdd9d700 1 -- 172.21.15.39:0/1411626813 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37c811a320 con 0x7f37c81026c0 2023-06-22T17:21:18.870 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.861+0000 7f37cdd9d700 1 -- 172.21.15.39:0/1411626813 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37c805f060 con 0x7f37c81026c0 2023-06-22T17:21:18.871 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.861+0000 7f37bffff700 1 -- 172.21.15.39:0/1411626813 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f37b801c640 con 0x7f37c81026c0 2023-06-22T17:21:18.871 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.861+0000 7f37bffff700 1 --2- 172.21.15.39:0/1411626813 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f37b4036b20 0x7f37b4038fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:21:18.871 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.861+0000 7f37bffff700 1 -- 172.21.15.39:0/1411626813 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f37b8049630 con 0x7f37c81026c0 2023-06-22T17:21:18.871 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.861+0000 7f37c6ffd700 1 --2- 172.21.15.39:0/1411626813 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f37b4036b20 0x7f37b4038fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:21:18.871 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.862+0000 7f37c6ffd700 1 --2- 172.21.15.39:0/1411626813 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f37b4036b20 0x7f37b4038fd0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f37b0009a00 tx=0x7f37b0008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:21:18.872 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:18.866+0000 7f37bffff700 1 -- 172.21.15.39:0/1411626813 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f37b8024950 con 0x7f37c81026c0 2023-06-22T17:21:19.069 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:19.068+0000 7f37cdd9d700 1 -- 172.21.15.39:0/1411626813 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f37c811a510 con 0x7f37c81026c0 2023-06-22T17:21:19.069 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:19.069+0000 7f37bffff700 1 -- 172.21.15.39:0/1411626813 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+275 (secure 0 0 0) 0x7f37b801b390 con 0x7f37c81026c0 2023-06-22T17:21:19.070 INFO:teuthology.orchestra.run.smithi039.stdout:# minimal ceph.conf for d88002e2-1120-11ee-9b2b-001a4aab830c 2023-06-22T17:21:19.070 INFO:teuthology.orchestra.run.smithi039.stdout:[global] 2023-06-22T17:21:19.070 INFO:teuthology.orchestra.run.smithi039.stdout: fsid = d88002e2-1120-11ee-9b2b-001a4aab830c 2023-06-22T17:21:19.070 INFO:teuthology.orchestra.run.smithi039.stdout: mon_host = [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] 2023-06-22T17:21:19.071 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:19.071+0000 7f37cdd9d700 1 -- 172.21.15.39:0/1411626813 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f37b4036b20 msgr2=0x7f37b4038fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:21:19.072 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:19.071+0000 7f37cdd9d700 1 --2- 172.21.15.39:0/1411626813 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f37b4036b20 0x7f37b4038fd0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f37b0009a00 tx=0x7f37b0008040 comp rx=0 tx=0).stop 2023-06-22T17:21:19.072 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:19.071+0000 7f37cdd9d700 1 -- 172.21.15.39:0/1411626813 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37c81026c0 msgr2=0x7f37c8112ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:21:19.072 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:19.071+0000 7f37cdd9d700 1 --2- 172.21.15.39:0/1411626813 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37c81026c0 0x7f37c8112ce0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f37b8008fe0 tx=0x7f37b8009e30 comp rx=0 tx=0).stop 2023-06-22T17:21:19.072 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:19.071+0000 7f37cdd9d700 1 -- 172.21.15.39:0/1411626813 shutdown_connections 2023-06-22T17:21:19.072 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:19.071+0000 7f37cdd9d700 1 --2- 172.21.15.39:0/1411626813 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f37b4036b20 0x7f37b4038fd0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:19.072 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:19.071+0000 7f37cdd9d700 1 --2- 172.21.15.39:0/1411626813 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f37c810d0c0 0x7f37c810d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:19.072 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:19.071+0000 7f37cdd9d700 1 --2- 172.21.15.39:0/1411626813 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37c81026c0 0x7f37c8112ce0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:19.073 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:19.071+0000 7f37cdd9d700 1 --2- 172.21.15.39:0/1411626813 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f37c8113220 0x7f37c810cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:19.073 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:19.071+0000 7f37cdd9d700 1 -- 172.21.15.39:0/1411626813 >> 172.21.15.39:0/1411626813 conn(0x7f37c80f81e0 msgr2=0x7f37c80f85b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:21:19.073 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:19.071+0000 7f37cdd9d700 1 -- 172.21.15.39:0/1411626813 shutdown_connections 2023-06-22T17:21:19.073 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:19.071+0000 7f37cdd9d700 1 -- 172.21.15.39:0/1411626813 wait complete. 2023-06-22T17:21:19.278 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:19 smithi039 ceph-mon[84805]: mon.b calling monitor election 2023-06-22T17:21:19.278 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:19 smithi039 ceph-mon[84805]: Updating smithi039:/etc/ceph/ceph.conf 2023-06-22T17:21:19.278 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:19 smithi039 ceph-mon[84805]: Updating smithi101:/etc/ceph/ceph.conf 2023-06-22T17:21:19.278 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:19 smithi039 ceph-mon[84805]: Updating smithi133:/etc/ceph/ceph.conf 2023-06-22T17:21:19.279 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:19 smithi039 ceph-mon[84805]: mon.b calling monitor election 2023-06-22T17:21:19.279 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:19 smithi039 ceph-mon[84805]: mon.c calling monitor election 2023-06-22T17:21:19.279 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:19 smithi039 ceph-mon[84805]: mon.a calling monitor election 2023-06-22T17:21:19.279 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:19 smithi039 ceph-mon[84805]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-06-22T17:21:19.279 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:19 smithi039 ceph-mon[84805]: monmap e3: 3 mons at {a=[v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0],b=[v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0],c=[v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0]} removed_ranks: {} 2023-06-22T17:21:19.279 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:19 smithi039 ceph-mon[84805]: Updating smithi101:/var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/config/ceph.conf 2023-06-22T17:21:19.279 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:19 smithi039 ceph-mon[84805]: fsmap 2023-06-22T17:21:19.279 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:19 smithi039 ceph-mon[84805]: osdmap e4: 0 total, 0 up, 0 in 2023-06-22T17:21:19.280 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:19 smithi039 ceph-mon[84805]: mgrmap e13: x(active, since 50s) 2023-06-22T17:21:19.280 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:19 smithi039 ceph-mon[84805]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-06-22T17:21:19.280 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:19 smithi039 ceph-mon[84805]: Cluster is now healthy 2023-06-22T17:21:19.280 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:19 smithi039 ceph-mon[84805]: overall HEALTH_OK 2023-06-22T17:21:19.280 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:19 smithi039 ceph-mon[84805]: Updating smithi133:/var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/config/ceph.conf 2023-06-22T17:21:19.280 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:19 smithi039 ceph-mon[84805]: Updating smithi039:/var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/config/ceph.conf 2023-06-22T17:21:19.281 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:19 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:19.281 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:19 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:19.281 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:19 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:19.281 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:19 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:19.281 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:19 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:19.282 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:19 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:19.282 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:19 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:19.282 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:19 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-06-22T17:21:19.282 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:19 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-06-22T17:21:19.282 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:19 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:19.375 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:19 smithi101 ceph-mon[85507]: mon.b calling monitor election 2023-06-22T17:21:19.375 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:19 smithi101 ceph-mon[85507]: Updating smithi039:/etc/ceph/ceph.conf 2023-06-22T17:21:19.375 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:19 smithi101 ceph-mon[85507]: Updating smithi101:/etc/ceph/ceph.conf 2023-06-22T17:21:19.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:19 smithi101 ceph-mon[85507]: Updating smithi133:/etc/ceph/ceph.conf 2023-06-22T17:21:19.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:19 smithi101 ceph-mon[85507]: mon.b calling monitor election 2023-06-22T17:21:19.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:19 smithi101 ceph-mon[85507]: mon.c calling monitor election 2023-06-22T17:21:19.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:19 smithi101 ceph-mon[85507]: mon.a calling monitor election 2023-06-22T17:21:19.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:19 smithi101 ceph-mon[85507]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-06-22T17:21:19.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:19 smithi101 ceph-mon[85507]: monmap e3: 3 mons at {a=[v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0],b=[v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0],c=[v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0]} removed_ranks: {} 2023-06-22T17:21:19.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:19 smithi101 ceph-mon[85507]: Updating smithi101:/var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/config/ceph.conf 2023-06-22T17:21:19.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:19 smithi101 ceph-mon[85507]: fsmap 2023-06-22T17:21:19.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:19 smithi101 ceph-mon[85507]: osdmap e4: 0 total, 0 up, 0 in 2023-06-22T17:21:19.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:19 smithi101 ceph-mon[85507]: mgrmap e13: x(active, since 50s) 2023-06-22T17:21:19.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:19 smithi101 ceph-mon[85507]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-06-22T17:21:19.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:19 smithi101 ceph-mon[85507]: Cluster is now healthy 2023-06-22T17:21:19.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:19 smithi101 ceph-mon[85507]: overall HEALTH_OK 2023-06-22T17:21:19.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:19 smithi101 ceph-mon[85507]: Updating smithi133:/var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/config/ceph.conf 2023-06-22T17:21:19.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:19 smithi101 ceph-mon[85507]: Updating smithi039:/var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/config/ceph.conf 2023-06-22T17:21:19.378 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:19 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:19.378 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:19 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:19.378 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:19 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:19.378 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:19 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:19.378 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:19 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:19.378 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:19 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:19.378 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:19 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:19.378 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:19 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-06-22T17:21:19.379 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:19 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-06-22T17:21:19.379 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:19 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:19.419 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:19 smithi133 ceph-mon[86268]: mon.b calling monitor election 2023-06-22T17:21:19.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:19 smithi133 ceph-mon[86268]: Updating smithi039:/etc/ceph/ceph.conf 2023-06-22T17:21:19.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:19 smithi133 ceph-mon[86268]: Updating smithi101:/etc/ceph/ceph.conf 2023-06-22T17:21:19.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:19 smithi133 ceph-mon[86268]: Updating smithi133:/etc/ceph/ceph.conf 2023-06-22T17:21:19.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:19 smithi133 ceph-mon[86268]: mon.b calling monitor election 2023-06-22T17:21:19.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:19 smithi133 ceph-mon[86268]: mon.c calling monitor election 2023-06-22T17:21:19.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:19 smithi133 ceph-mon[86268]: mon.a calling monitor election 2023-06-22T17:21:19.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:19 smithi133 ceph-mon[86268]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-06-22T17:21:19.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:19 smithi133 ceph-mon[86268]: monmap e3: 3 mons at {a=[v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0],b=[v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0],c=[v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0]} removed_ranks: {} 2023-06-22T17:21:19.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:19 smithi133 ceph-mon[86268]: Updating smithi101:/var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/config/ceph.conf 2023-06-22T17:21:19.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:19 smithi133 ceph-mon[86268]: fsmap 2023-06-22T17:21:19.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:19 smithi133 ceph-mon[86268]: osdmap e4: 0 total, 0 up, 0 in 2023-06-22T17:21:19.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:19 smithi133 ceph-mon[86268]: mgrmap e13: x(active, since 50s) 2023-06-22T17:21:19.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:19 smithi133 ceph-mon[86268]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-06-22T17:21:19.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:19 smithi133 ceph-mon[86268]: Cluster is now healthy 2023-06-22T17:21:19.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:19 smithi133 ceph-mon[86268]: overall HEALTH_OK 2023-06-22T17:21:19.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:19 smithi133 ceph-mon[86268]: Updating smithi133:/var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/config/ceph.conf 2023-06-22T17:21:19.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:19 smithi133 ceph-mon[86268]: Updating smithi039:/var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/config/ceph.conf 2023-06-22T17:21:19.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:19 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:19.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:19 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:19.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:19 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:19.423 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:19 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:19.423 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:19 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:19.423 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:19 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:19.423 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:19 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:19.423 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:19 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-06-22T17:21:19.423 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:19 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-06-22T17:21:19.423 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:19 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:19.705 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-06-22T17:21:19.705 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-22T17:21:19.706 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/ceph/ceph.conf 2023-06-22T17:21:19.734 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-22T17:21:19.734 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-06-22T17:21:19.798 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-06-22T17:21:19.799 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/etc/ceph/ceph.conf 2023-06-22T17:21:19.829 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-06-22T17:21:19.830 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-06-22T17:21:19.895 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-06-22T17:21:19.895 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/ceph/ceph.conf 2023-06-22T17:21:19.925 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-06-22T17:21:19.926 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-06-22T17:21:19.992 INFO:tasks.cephadm:Adding mgr.x on smithi039 2023-06-22T17:21:19.992 INFO:tasks.cephadm:Adding mgr.y on smithi101 2023-06-22T17:21:19.992 INFO:tasks.cephadm:Adding mgr.z on smithi133 2023-06-22T17:21:19.993 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph orch apply mgr '3;smithi039=x;smithi101=y;smithi133=z' 2023-06-22T17:21:20.125 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:20 smithi101 ceph-mon[85507]: Reconfiguring mon.a (unknown last config time)... 2023-06-22T17:21:20.125 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:20 smithi101 ceph-mon[85507]: Reconfiguring daemon mon.a on smithi039 2023-06-22T17:21:20.125 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:20 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/1411626813' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:20.125 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:20 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-22T17:21:20.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:20 smithi101 ceph-mon[85507]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:20.419 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:20 smithi133 ceph-mon[86268]: Reconfiguring mon.a (unknown last config time)... 2023-06-22T17:21:20.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:20 smithi133 ceph-mon[86268]: Reconfiguring daemon mon.a on smithi039 2023-06-22T17:21:20.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:20 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/1411626813' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:20.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:20 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-22T17:21:20.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:20 smithi133 ceph-mon[86268]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:20.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:20 smithi039 ceph-mon[84805]: Reconfiguring mon.a (unknown last config time)... 2023-06-22T17:21:20.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:20 smithi039 ceph-mon[84805]: Reconfiguring daemon mon.a on smithi039 2023-06-22T17:21:20.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:20 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/1411626813' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:20.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:20 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-22T17:21:20.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:20 smithi039 ceph-mon[84805]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:20.608 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.c/config 2023-06-22T17:21:21.375 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:21 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:21.375 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:21 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:21.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:21 smithi101 ceph-mon[85507]: Reconfiguring mon.b (monmap changed)... 2023-06-22T17:21:21.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:21 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-06-22T17:21:21.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:21 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-06-22T17:21:21.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:21 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:21.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:21 smithi101 ceph-mon[85507]: Reconfiguring daemon mon.b on smithi101 2023-06-22T17:21:21.419 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:21 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:21.419 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:21 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:21.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:21 smithi133 ceph-mon[86268]: Reconfiguring mon.b (monmap changed)... 2023-06-22T17:21:21.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:21 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-06-22T17:21:21.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:21 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-06-22T17:21:21.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:21 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:21.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:21 smithi133 ceph-mon[86268]: Reconfiguring daemon mon.b on smithi101 2023-06-22T17:21:21.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:21 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:21.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:21 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:21.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:21 smithi039 ceph-mon[84805]: Reconfiguring mon.b (monmap changed)... 2023-06-22T17:21:21.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:21 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-06-22T17:21:21.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:21 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-06-22T17:21:21.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:21 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:21.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:21 smithi039 ceph-mon[84805]: Reconfiguring daemon mon.b on smithi101 2023-06-22T17:21:21.834 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.833+0000 7f8991504700 1 -- 172.21.15.133:0/2061186735 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f898c065de0 msgr2=0x7f898c0ff380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:21:21.835 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.833+0000 7f8991504700 1 --2- 172.21.15.133:0/2061186735 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f898c065de0 0x7f898c0ff380 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f897c0048c0 tx=0x7f897c009e30 comp rx=0 tx=0).stop 2023-06-22T17:21:21.835 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.833+0000 7f8991504700 1 -- 172.21.15.133:0/2061186735 shutdown_connections 2023-06-22T17:21:21.835 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.833+0000 7f8991504700 1 --2- 172.21.15.133:0/2061186735 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f898c065de0 0x7f898c0ff380 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:21.836 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.833+0000 7f8991504700 1 -- 172.21.15.133:0/2061186735 >> 172.21.15.133:0/2061186735 conn(0x7f898c0f6db0 msgr2=0x7f898c0f91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:21:21.836 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.833+0000 7f8991504700 1 -- 172.21.15.133:0/2061186735 shutdown_connections 2023-06-22T17:21:21.836 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.833+0000 7f8991504700 1 -- 172.21.15.133:0/2061186735 wait complete. 2023-06-22T17:21:21.840 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.834+0000 7f8991504700 1 Processor -- start 2023-06-22T17:21:21.840 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.834+0000 7f8991504700 1 -- start start 2023-06-22T17:21:21.841 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.834+0000 7f8991504700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f898c065de0 0x7f898c10cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:21:21.841 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.834+0000 7f8991504700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f898c10d2a0 0x7f898c10e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:21:21.841 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.834+0000 7f8991504700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f898c10e5a0 0x7f898c10ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:21:21.841 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.834+0000 7f8991504700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898c1152b0 con 0x7f898c10e5a0 2023-06-22T17:21:21.842 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.834+0000 7f8991504700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898c1153f0 con 0x7f898c10d2a0 2023-06-22T17:21:21.842 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.834+0000 7f8991504700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898c115530 con 0x7f898c065de0 2023-06-22T17:21:21.842 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.834+0000 7f898affd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f898c065de0 0x7f898c10cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:21:21.843 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.834+0000 7f898affd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f898c065de0 0x7f898c10cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:43880/0 (socket says 172.21.15.133:43880) 2023-06-22T17:21:21.843 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.834+0000 7f898affd700 1 -- 172.21.15.133:0/3205318915 learned_addr learned my addr 172.21.15.133:0/3205318915 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-06-22T17:21:21.843 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.834+0000 7f898b7fe700 1 --2- 172.21.15.133:0/3205318915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f898c10e5a0 0x7f898c10ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:21:21.844 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.835+0000 7f898a7fc700 1 --2- 172.21.15.133:0/3205318915 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f898c10d2a0 0x7f898c10e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:21:21.844 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.835+0000 7f898affd700 1 -- 172.21.15.133:0/3205318915 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f898c10d2a0 msgr2=0x7f898c10e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:21:21.844 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.835+0000 7f898affd700 1 --2- 172.21.15.133:0/3205318915 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f898c10d2a0 0x7f898c10e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:21.845 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.835+0000 7f898affd700 1 -- 172.21.15.133:0/3205318915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f898c10e5a0 msgr2=0x7f898c10ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:21:21.845 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.835+0000 7f898affd700 1 --2- 172.21.15.133:0/3205318915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f898c10e5a0 0x7f898c10ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:21.845 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.835+0000 7f898affd700 1 -- 172.21.15.133:0/3205318915 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f897c005040 con 0x7f898c065de0 2023-06-22T17:21:21.845 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.835+0000 7f898a7fc700 1 --2- 172.21.15.133:0/3205318915 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f898c10d2a0 0x7f898c10e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-22T17:21:21.846 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.835+0000 7f898affd700 1 --2- 172.21.15.133:0/3205318915 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f898c065de0 0x7f898c10cd60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f897c004390 tx=0x7f897c009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:21:21.846 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.835+0000 7f89737fe700 1 -- 172.21.15.133:0/3205318915 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f897c00e040 con 0x7f898c065de0 2023-06-22T17:21:21.846 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.835+0000 7f89737fe700 1 -- 172.21.15.133:0/3205318915 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f897c0098e0 con 0x7f898c065de0 2023-06-22T17:21:21.847 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.835+0000 7f8991504700 1 -- 172.21.15.133:0/3205318915 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f898c115760 con 0x7f898c065de0 2023-06-22T17:21:21.847 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.835+0000 7f89737fe700 1 -- 172.21.15.133:0/3205318915 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f897c01c3f0 con 0x7f898c065de0 2023-06-22T17:21:21.847 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.835+0000 7f8991504700 1 -- 172.21.15.133:0/3205318915 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f898c115c70 con 0x7f898c065de0 2023-06-22T17:21:21.848 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.836+0000 7f89737fe700 1 -- 172.21.15.133:0/3205318915 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 13) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f897c01c820 con 0x7f898c065de0 2023-06-22T17:21:21.848 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.836+0000 7f89737fe700 1 --2- 172.21.15.133:0/3205318915 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f8974036ad0 0x7f8974038f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:21:21.848 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.836+0000 7f89737fe700 1 -- 172.21.15.133:0/3205318915 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f897c049850 con 0x7f898c065de0 2023-06-22T17:21:21.848 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.836+0000 7f8991504700 1 -- 172.21.15.133:0/3205318915 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f898c106500 con 0x7f898c065de0 2023-06-22T17:21:21.849 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.836+0000 7f898a7fc700 1 --2- 172.21.15.133:0/3205318915 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f8974036ad0 0x7f8974038f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:21:21.849 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.837+0000 7f898a7fc700 1 --2- 172.21.15.133:0/3205318915 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f8974036ad0 0x7f8974038f80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f89800079f0 tx=0x7f8980008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:21:21.850 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:21.841+0000 7f89737fe700 1 -- 172.21.15.133:0/3205318915 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f897c009400 con 0x7f898c065de0 2023-06-22T17:21:22.050 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:22.049+0000 7f8991504700 1 -- 172.21.15.133:0/3205318915 --> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi039=x;smithi101=y;smithi133=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f898c05de70 con 0x7f8974036ad0 2023-06-22T17:21:22.061 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:22.060+0000 7f89737fe700 1 -- 172.21.15.133:0/3205318915 <== mgr.14150 v2:172.21.15.39:6800/886078282 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f898c05de70 con 0x7f8974036ad0 2023-06-22T17:21:22.061 INFO:teuthology.orchestra.run.smithi133.stdout:Scheduled mgr update... 2023-06-22T17:21:22.063 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:22.062+0000 7f8991504700 1 -- 172.21.15.133:0/3205318915 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f8974036ad0 msgr2=0x7f8974038f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:21:22.063 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:22.062+0000 7f8991504700 1 --2- 172.21.15.133:0/3205318915 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f8974036ad0 0x7f8974038f80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f89800079f0 tx=0x7f8980008040 comp rx=0 tx=0).stop 2023-06-22T17:21:22.063 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:22.062+0000 7f8991504700 1 -- 172.21.15.133:0/3205318915 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f898c065de0 msgr2=0x7f898c10cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:21:22.064 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:22.062+0000 7f8991504700 1 --2- 172.21.15.133:0/3205318915 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f898c065de0 0x7f898c10cd60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f897c004390 tx=0x7f897c009e30 comp rx=0 tx=0).stop 2023-06-22T17:21:22.064 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:22.062+0000 7f8991504700 1 -- 172.21.15.133:0/3205318915 shutdown_connections 2023-06-22T17:21:22.064 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:22.062+0000 7f8991504700 1 --2- 172.21.15.133:0/3205318915 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f8974036ad0 0x7f8974038f80 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:22.064 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:22.062+0000 7f8991504700 1 --2- 172.21.15.133:0/3205318915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f898c10e5a0 0x7f898c10ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:22.064 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:22.062+0000 7f8991504700 1 --2- 172.21.15.133:0/3205318915 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f898c065de0 0x7f898c10cd60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:22.064 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:22.062+0000 7f8991504700 1 --2- 172.21.15.133:0/3205318915 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f898c10d2a0 0x7f898c10e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:22.065 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:22.062+0000 7f8991504700 1 -- 172.21.15.133:0/3205318915 >> 172.21.15.133:0/3205318915 conn(0x7f898c0f6db0 msgr2=0x7f898c0f7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:21:22.065 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:22.062+0000 7f8991504700 1 -- 172.21.15.133:0/3205318915 shutdown_connections 2023-06-22T17:21:22.065 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:21:22.062+0000 7f8991504700 1 -- 172.21.15.133:0/3205318915 wait complete. 2023-06-22T17:21:22.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:22 smithi101 ceph-mon[85507]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:22.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:22 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:22.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:22 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:22.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:22 smithi101 ceph-mon[85507]: Reconfiguring mon.c (monmap changed)... 2023-06-22T17:21:22.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:22 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-06-22T17:21:22.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:22 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-06-22T17:21:22.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:22 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:22.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:22 smithi101 ceph-mon[85507]: Reconfiguring daemon mon.c on smithi133 2023-06-22T17:21:22.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:22 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:22.655 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:22 smithi133 ceph-mon[86268]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:22.655 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:22 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:22.655 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:22 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:22.656 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:22 smithi133 ceph-mon[86268]: Reconfiguring mon.c (monmap changed)... 2023-06-22T17:21:22.656 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:22 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-06-22T17:21:22.656 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:22 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-06-22T17:21:22.656 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:22 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:22.656 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:22 smithi133 ceph-mon[86268]: Reconfiguring daemon mon.c on smithi133 2023-06-22T17:21:22.656 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:22 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:22.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:22 smithi039 ceph-mon[84805]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:22.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:22 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:22.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:22 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:22.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:22 smithi039 ceph-mon[84805]: Reconfiguring mon.c (monmap changed)... 2023-06-22T17:21:22.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:22 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-06-22T17:21:22.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:22 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-06-22T17:21:22.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:22 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:22.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:22 smithi039 ceph-mon[84805]: Reconfiguring daemon mon.c on smithi133 2023-06-22T17:21:22.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:22 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:22.862 DEBUG:teuthology.orchestra.run.smithi101:mgr.y> sudo journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mgr.y.service 2023-06-22T17:21:22.865 DEBUG:teuthology.orchestra.run.smithi133:mgr.z> sudo journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mgr.z.service 2023-06-22T17:21:22.867 INFO:tasks.cephadm:Deploying OSDs... 2023-06-22T17:21:22.867 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-22T17:21:22.867 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/scratch_devs of=/dev/stdout 2023-06-22T17:21:22.886 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-06-22T17:21:22.886 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_1 2023-06-22T17:21:22.893 INFO:journalctl@ceph.mgr.z.smithi133.stdout:-- Logs begin at Thu 2023-06-22 17:06:34 UTC. -- 2023-06-22T17:21:22.894 INFO:journalctl@ceph.mgr.y.smithi101.stdout:-- Logs begin at Thu 2023-06-22 17:06:34 UTC. -- 2023-06-22T17:21:22.945 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-06-22T17:21:22.945 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-22T17:21:22.945 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 6h/6d Inode: 214099 Links: 1 2023-06-22T17:21:22.945 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-22T17:21:22.946 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2023-06-22T17:21:22.946 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2023-06-22 17:20:37.828922167 +0000 2023-06-22T17:21:22.946 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2023-06-22 17:15:44.734530310 +0000 2023-06-22T17:21:22.946 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2023-06-22 17:15:44.734530310 +0000 2023-06-22T17:21:22.946 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: - 2023-06-22T17:21:22.946 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-06-22T17:21:23.016 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2023-06-22T17:21:23.016 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2023-06-22T17:21:23.017 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000396643 s, 1.3 MB/s 2023-06-22T17:21:23.017 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-06-22T17:21:23.076 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_2 2023-06-22T17:21:23.133 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-06-22T17:21:23.134 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-22T17:21:23.134 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 6h/6d Inode: 209598 Links: 1 2023-06-22T17:21:23.134 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-22T17:21:23.134 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2023-06-22T17:21:23.134 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2023-06-22 17:20:37.828922167 +0000 2023-06-22T17:21:23.134 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2023-06-22 17:15:45.245519907 +0000 2023-06-22T17:21:23.135 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2023-06-22 17:15:45.245519907 +0000 2023-06-22T17:21:23.135 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: - 2023-06-22T17:21:23.135 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-06-22T17:21:23.200 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2023-06-22T17:21:23.201 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2023-06-22T17:21:23.201 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000244904 s, 2.1 MB/s 2023-06-22T17:21:23.202 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-06-22T17:21:23.260 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_3 2023-06-22T17:21:23.316 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-06-22T17:21:23.316 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-22T17:21:23.316 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 6h/6d Inode: 209718 Links: 1 2023-06-22T17:21:23.316 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-22T17:21:23.316 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2023-06-22T17:21:23.317 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2023-06-22 17:20:37.829922147 +0000 2023-06-22T17:21:23.317 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2023-06-22 17:15:45.770509217 +0000 2023-06-22T17:21:23.317 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2023-06-22 17:15:45.770509217 +0000 2023-06-22T17:21:23.317 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: - 2023-06-22T17:21:23.317 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-06-22T17:21:23.382 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2023-06-22T17:21:23.382 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2023-06-22T17:21:23.383 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000282967 s, 1.8 MB/s 2023-06-22T17:21:23.384 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-06-22T17:21:23.441 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_4 2023-06-22T17:21:23.497 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-06-22T17:21:23.498 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-22T17:21:23.498 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 6h/6d Inode: 211488 Links: 1 2023-06-22T17:21:23.498 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-22T17:21:23.498 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2023-06-22T17:21:23.499 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2023-06-22 17:20:37.829922147 +0000 2023-06-22T17:21:23.499 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2023-06-22 17:15:46.287498692 +0000 2023-06-22T17:21:23.499 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2023-06-22 17:15:46.287498692 +0000 2023-06-22T17:21:23.499 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: - 2023-06-22T17:21:23.499 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-06-22T17:21:23.562 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2023-06-22T17:21:23.562 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2023-06-22T17:21:23.562 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000296424 s, 1.7 MB/s 2023-06-22T17:21:23.563 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-06-22T17:21:23.620 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-06-22T17:21:23.620 DEBUG:teuthology.orchestra.run.smithi101:> dd if=/scratch_devs of=/dev/stdout 2023-06-22T17:21:23.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:23 smithi101 ceph-mon[85507]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi039=x;smithi101=y;smithi133=z", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:21:23.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:23 smithi101 ceph-mon[85507]: Saving service mgr spec with placement smithi039=x;smithi101=y;smithi133=z;count:3 2023-06-22T17:21:23.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:23 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:23.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:23 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:23.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:23 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:21:23.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:23 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:23.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:23 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:21:23.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:23 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:23.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:23 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-06-22T17:21:23.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:23 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-06-22T17:21:23.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:23 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-06-22T17:21:23.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:23 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:23.638 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-06-22T17:21:23.639 DEBUG:teuthology.orchestra.run.smithi101:> stat /dev/vg_nvme/lv_1 2023-06-22T17:21:23.669 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:23 smithi133 ceph-mon[86268]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi039=x;smithi101=y;smithi133=z", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:21:23.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:23 smithi133 ceph-mon[86268]: Saving service mgr spec with placement smithi039=x;smithi101=y;smithi133=z;count:3 2023-06-22T17:21:23.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:23 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:23.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:23 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:23.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:23 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:21:23.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:23 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:23.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:23 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:21:23.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:23 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:23.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:23 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-06-22T17:21:23.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:23 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-06-22T17:21:23.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:23 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-06-22T17:21:23.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:23 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:23.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:23 smithi039 ceph-mon[84805]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi039=x;smithi101=y;smithi133=z", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:21:23.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:23 smithi039 ceph-mon[84805]: Saving service mgr spec with placement smithi039=x;smithi101=y;smithi133=z;count:3 2023-06-22T17:21:23.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:23 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:23.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:23 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:23.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:23 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:21:23.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:23 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:23.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:23 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:21:23.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:23 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:23.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:23 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-06-22T17:21:23.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:23 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-06-22T17:21:23.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:23 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-06-22T17:21:23.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:23 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:23.696 INFO:teuthology.orchestra.run.smithi101.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-06-22T17:21:23.696 INFO:teuthology.orchestra.run.smithi101.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-22T17:21:23.696 INFO:teuthology.orchestra.run.smithi101.stdout:Device: 6h/6d Inode: 212310 Links: 1 2023-06-22T17:21:23.697 INFO:teuthology.orchestra.run.smithi101.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-22T17:21:23.697 INFO:teuthology.orchestra.run.smithi101.stdout:Context: system_u:object_r:device_t:s0 2023-06-22T17:21:23.697 INFO:teuthology.orchestra.run.smithi101.stdout:Access: 2023-06-22 17:21:10.566515050 +0000 2023-06-22T17:21:23.697 INFO:teuthology.orchestra.run.smithi101.stdout:Modify: 2023-06-22 17:15:32.401388003 +0000 2023-06-22T17:21:23.697 INFO:teuthology.orchestra.run.smithi101.stdout:Change: 2023-06-22 17:15:32.401388003 +0000 2023-06-22T17:21:23.697 INFO:teuthology.orchestra.run.smithi101.stdout: Birth: - 2023-06-22T17:21:23.698 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-06-22T17:21:23.762 INFO:teuthology.orchestra.run.smithi101.stderr:1+0 records in 2023-06-22T17:21:23.762 INFO:teuthology.orchestra.run.smithi101.stderr:1+0 records out 2023-06-22T17:21:23.762 INFO:teuthology.orchestra.run.smithi101.stderr:512 bytes copied, 0.000262304 s, 2.0 MB/s 2023-06-22T17:21:23.763 DEBUG:teuthology.orchestra.run.smithi101:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-06-22T17:21:23.821 DEBUG:teuthology.orchestra.run.smithi101:> stat /dev/vg_nvme/lv_2 2023-06-22T17:21:23.877 INFO:teuthology.orchestra.run.smithi101.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-06-22T17:21:23.877 INFO:teuthology.orchestra.run.smithi101.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-22T17:21:23.878 INFO:teuthology.orchestra.run.smithi101.stdout:Device: 6h/6d Inode: 210484 Links: 1 2023-06-22T17:21:23.878 INFO:teuthology.orchestra.run.smithi101.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-22T17:21:23.878 INFO:teuthology.orchestra.run.smithi101.stdout:Context: system_u:object_r:device_t:s0 2023-06-22T17:21:23.878 INFO:teuthology.orchestra.run.smithi101.stdout:Access: 2023-06-22 17:21:10.566515050 +0000 2023-06-22T17:21:23.879 INFO:teuthology.orchestra.run.smithi101.stdout:Modify: 2023-06-22 17:15:32.893376483 +0000 2023-06-22T17:21:23.879 INFO:teuthology.orchestra.run.smithi101.stdout:Change: 2023-06-22 17:15:32.893376483 +0000 2023-06-22T17:21:23.879 INFO:teuthology.orchestra.run.smithi101.stdout: Birth: - 2023-06-22T17:21:23.880 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-06-22T17:21:23.944 INFO:teuthology.orchestra.run.smithi101.stderr:1+0 records in 2023-06-22T17:21:23.944 INFO:teuthology.orchestra.run.smithi101.stderr:1+0 records out 2023-06-22T17:21:23.945 INFO:teuthology.orchestra.run.smithi101.stderr:512 bytes copied, 0.000263836 s, 1.9 MB/s 2023-06-22T17:21:23.946 DEBUG:teuthology.orchestra.run.smithi101:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-06-22T17:21:24.003 DEBUG:teuthology.orchestra.run.smithi101:> stat /dev/vg_nvme/lv_3 2023-06-22T17:21:24.060 INFO:teuthology.orchestra.run.smithi101.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-06-22T17:21:24.060 INFO:teuthology.orchestra.run.smithi101.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-22T17:21:24.061 INFO:teuthology.orchestra.run.smithi101.stdout:Device: 6h/6d Inode: 215132 Links: 1 2023-06-22T17:21:24.061 INFO:teuthology.orchestra.run.smithi101.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-22T17:21:24.061 INFO:teuthology.orchestra.run.smithi101.stdout:Context: system_u:object_r:device_t:s0 2023-06-22T17:21:24.061 INFO:teuthology.orchestra.run.smithi101.stdout:Access: 2023-06-22 17:21:10.566515050 +0000 2023-06-22T17:21:24.061 INFO:teuthology.orchestra.run.smithi101.stdout:Modify: 2023-06-22 17:15:33.394364752 +0000 2023-06-22T17:21:24.061 INFO:teuthology.orchestra.run.smithi101.stdout:Change: 2023-06-22 17:15:33.394364752 +0000 2023-06-22T17:21:24.062 INFO:teuthology.orchestra.run.smithi101.stdout: Birth: - 2023-06-22T17:21:24.062 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-06-22T17:21:24.124 INFO:teuthology.orchestra.run.smithi101.stderr:1+0 records in 2023-06-22T17:21:24.124 INFO:teuthology.orchestra.run.smithi101.stderr:1+0 records out 2023-06-22T17:21:24.124 INFO:teuthology.orchestra.run.smithi101.stderr:512 bytes copied, 0.000248409 s, 2.1 MB/s 2023-06-22T17:21:24.125 DEBUG:teuthology.orchestra.run.smithi101:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-06-22T17:21:24.182 DEBUG:teuthology.orchestra.run.smithi101:> stat /dev/vg_nvme/lv_4 2023-06-22T17:21:24.238 INFO:teuthology.orchestra.run.smithi101.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-06-22T17:21:24.238 INFO:teuthology.orchestra.run.smithi101.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-22T17:21:24.239 INFO:teuthology.orchestra.run.smithi101.stdout:Device: 6h/6d Inode: 213466 Links: 1 2023-06-22T17:21:24.239 INFO:teuthology.orchestra.run.smithi101.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-22T17:21:24.239 INFO:teuthology.orchestra.run.smithi101.stdout:Context: system_u:object_r:device_t:s0 2023-06-22T17:21:24.239 INFO:teuthology.orchestra.run.smithi101.stdout:Access: 2023-06-22 17:21:10.567515028 +0000 2023-06-22T17:21:24.239 INFO:teuthology.orchestra.run.smithi101.stdout:Modify: 2023-06-22 17:15:33.894353105 +0000 2023-06-22T17:21:24.239 INFO:teuthology.orchestra.run.smithi101.stdout:Change: 2023-06-22 17:15:33.894353105 +0000 2023-06-22T17:21:24.239 INFO:teuthology.orchestra.run.smithi101.stdout: Birth: - 2023-06-22T17:21:24.240 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-06-22T17:21:24.303 INFO:teuthology.orchestra.run.smithi101.stderr:1+0 records in 2023-06-22T17:21:24.303 INFO:teuthology.orchestra.run.smithi101.stderr:1+0 records out 2023-06-22T17:21:24.303 INFO:teuthology.orchestra.run.smithi101.stderr:512 bytes copied, 0.000213836 s, 2.4 MB/s 2023-06-22T17:21:24.304 DEBUG:teuthology.orchestra.run.smithi101:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-06-22T17:21:24.344 INFO:journalctl@ceph.mgr.z.smithi133.stdout:Jun 22 17:21:24 smithi133 systemd[1]: Starting Ceph mgr.z for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:21:24.360 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-06-22T17:21:24.361 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/scratch_devs of=/dev/stdout 2023-06-22T17:21:24.375 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:24 smithi101 ceph-mon[85507]: Deploying daemon mgr.z on smithi133 2023-06-22T17:21:24.375 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:24 smithi101 ceph-mon[85507]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:24.377 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-06-22T17:21:24.377 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_1 2023-06-22T17:21:24.432 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-06-22T17:21:24.433 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-22T17:21:24.433 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 6h/6d Inode: 207621 Links: 1 2023-06-22T17:21:24.433 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-22T17:21:24.434 INFO:teuthology.orchestra.run.smithi133.stdout:Context: system_u:object_r:device_t:s0 2023-06-22T17:21:24.434 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2023-06-22 17:21:24.068827417 +0000 2023-06-22T17:21:24.434 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2023-06-22 17:15:33.288072214 +0000 2023-06-22T17:21:24.435 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2023-06-22 17:15:33.288072214 +0000 2023-06-22T17:21:24.435 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: - 2023-06-22T17:21:24.435 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-06-22T17:21:24.500 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2023-06-22T17:21:24.500 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2023-06-22T17:21:24.500 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000233425 s, 2.2 MB/s 2023-06-22T17:21:24.501 DEBUG:teuthology.orchestra.run.smithi133:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-06-22T17:21:24.557 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_2 2023-06-22T17:21:24.611 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-06-22T17:21:24.611 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-22T17:21:24.612 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 6h/6d Inode: 212908 Links: 1 2023-06-22T17:21:24.612 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-22T17:21:24.612 INFO:teuthology.orchestra.run.smithi133.stdout:Context: system_u:object_r:device_t:s0 2023-06-22T17:21:24.612 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2023-06-22 17:21:24.068827417 +0000 2023-06-22T17:21:24.612 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2023-06-22 17:15:33.770062927 +0000 2023-06-22T17:21:24.612 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2023-06-22 17:15:33.770062927 +0000 2023-06-22T17:21:24.613 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: - 2023-06-22T17:21:24.613 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-06-22T17:21:24.669 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:24 smithi133 ceph-mon[86268]: Deploying daemon mgr.z on smithi133 2023-06-22T17:21:24.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:24 smithi133 ceph-mon[86268]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:24.676 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2023-06-22T17:21:24.676 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2023-06-22T17:21:24.676 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000228446 s, 2.2 MB/s 2023-06-22T17:21:24.677 DEBUG:teuthology.orchestra.run.smithi133:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-06-22T17:21:24.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:24 smithi039 ceph-mon[84805]: Deploying daemon mgr.z on smithi133 2023-06-22T17:21:24.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:24 smithi039 ceph-mon[84805]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:24.734 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_3 2023-06-22T17:21:24.789 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-06-22T17:21:24.789 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-22T17:21:24.789 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 6h/6d Inode: 212970 Links: 1 2023-06-22T17:21:24.790 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-22T17:21:24.790 INFO:teuthology.orchestra.run.smithi133.stdout:Context: system_u:object_r:device_t:s0 2023-06-22T17:21:24.790 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2023-06-22 17:21:24.068827417 +0000 2023-06-22T17:21:24.790 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2023-06-22 17:15:34.271053276 +0000 2023-06-22T17:21:24.790 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2023-06-22 17:15:34.271053276 +0000 2023-06-22T17:21:24.790 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: - 2023-06-22T17:21:24.791 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-06-22T17:21:24.855 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2023-06-22T17:21:24.855 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2023-06-22T17:21:24.855 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000227683 s, 2.2 MB/s 2023-06-22T17:21:24.856 DEBUG:teuthology.orchestra.run.smithi133:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-06-22T17:21:24.914 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_4 2023-06-22T17:21:24.971 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-06-22T17:21:24.971 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-22T17:21:24.972 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 6h/6d Inode: 215138 Links: 1 2023-06-22T17:21:24.972 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-22T17:21:24.972 INFO:teuthology.orchestra.run.smithi133.stdout:Context: system_u:object_r:device_t:s0 2023-06-22T17:21:24.972 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2023-06-22 17:21:24.068827417 +0000 2023-06-22T17:21:24.972 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2023-06-22 17:15:34.780043470 +0000 2023-06-22T17:21:24.972 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2023-06-22 17:15:34.780043470 +0000 2023-06-22T17:21:24.973 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: - 2023-06-22T17:21:24.973 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-06-22T17:21:25.033 INFO:journalctl@ceph.mgr.z.smithi133.stdout:Jun 22 17:21:24 smithi133 bash[88882]: da409d0526eb962ae8ee4c4500c02d996c34cdd31699ff6a7b3512ed135ad844 2023-06-22T17:21:25.034 INFO:journalctl@ceph.mgr.z.smithi133.stdout:Jun 22 17:21:24 smithi133 ceph-mgr[89039]: -- 172.21.15.133:0/2242975846 <== mon.1 v2:172.21.15.133:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56397681a9c0 con 0x5639767f1000 2023-06-22T17:21:25.034 INFO:journalctl@ceph.mgr.z.smithi133.stdout:Jun 22 17:21:24 smithi133 systemd[1]: Started Ceph mgr.z for d88002e2-1120-11ee-9b2b-001a4aab830c. 2023-06-22T17:21:25.036 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2023-06-22T17:21:25.036 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2023-06-22T17:21:25.036 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000181568 s, 2.8 MB/s 2023-06-22T17:21:25.037 DEBUG:teuthology.orchestra.run.smithi133:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-06-22T17:21:25.093 INFO:tasks.cephadm:Deploying osd.0 on smithi039 with /dev/vg_nvme/lv_4... 2023-06-22T17:21:25.094 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-06-22T17:21:25.696 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-22T17:21:26.057 INFO:journalctl@ceph.mgr.z.smithi133.stdout:Jun 22 17:21:25 smithi133 conmon[88969]: 2023-06-22T17:21:25.898+0000 7fe81b8051c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-06-22T17:21:26.058 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:25 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:26.058 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:25 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:26.058 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:25 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:26.058 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:25 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-06-22T17:21:26.058 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:25 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-06-22T17:21:26.058 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:25 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-06-22T17:21:26.058 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:25 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:26.059 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:25 smithi133 ceph-mon[86268]: Deploying daemon mgr.y on smithi101 2023-06-22T17:21:26.059 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:25 smithi133 ceph-mon[86268]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:26.079 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:25 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:26.079 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:25 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:26.080 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:25 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:26.080 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:25 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-06-22T17:21:26.080 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:25 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-06-22T17:21:26.080 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:25 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-06-22T17:21:26.080 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:25 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:26.080 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:25 smithi039 ceph-mon[84805]: Deploying daemon mgr.y on smithi101 2023-06-22T17:21:26.081 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:25 smithi039 ceph-mon[84805]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:26.114 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:25 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:26.114 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:25 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:26.114 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:25 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:26.114 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:25 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-06-22T17:21:26.115 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:25 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-06-22T17:21:26.115 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:25 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-06-22T17:21:26.115 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:25 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:26.115 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:25 smithi101 ceph-mon[85507]: Deploying daemon mgr.y on smithi101 2023-06-22T17:21:26.115 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:25 smithi101 ceph-mon[85507]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:26.358 INFO:journalctl@ceph.mgr.z.smithi133.stdout:Jun 22 17:21:26 smithi133 conmon[88969]: 2023-06-22T17:21:26.056+0000 7fe81b8051c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-06-22T17:21:26.359 INFO:journalctl@ceph.mgr.z.smithi133.stdout:Jun 22 17:21:26 smithi133 conmon[88969]: 2023-06-22T17:21:26.144+0000 7fe81b8051c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-06-22T17:21:26.375 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jun 22 17:21:26 smithi101 systemd[1]: Starting Ceph mgr.y for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:21:26.669 INFO:journalctl@ceph.mgr.z.smithi133.stdout:Jun 22 17:21:26 smithi133 conmon[88969]: 2023-06-22T17:21:26.357+0000 7fe81b8051c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-06-22T17:21:26.669 INFO:journalctl@ceph.mgr.z.smithi133.stdout:Jun 22 17:21:26 smithi133 conmon[88969]: 2023-06-22T17:21:26.558+0000 7fe81b8051c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-06-22T17:21:27.125 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jun 22 17:21:26 smithi101 ceph-mgr[87188]: -- 172.21.15.101:0/484429950 <== mon.2 v2:172.21.15.101:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562a200909c0 con 0x562a20066800 2023-06-22T17:21:27.125 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jun 22 17:21:26 smithi101 bash[87123]: cf8db303a0680602e5cbc7012c94af2a947665fb01f70e7b8f911da39e77c18c 2023-06-22T17:21:27.125 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jun 22 17:21:26 smithi101 systemd[1]: Started Ceph mgr.y for d88002e2-1120-11ee-9b2b-001a4aab830c. 2023-06-22T17:21:27.169 INFO:journalctl@ceph.mgr.z.smithi133.stdout:Jun 22 17:21:26 smithi133 conmon[88969]: 2023-06-22T17:21:26.716+0000 7fe81b8051c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-06-22T17:21:27.169 INFO:journalctl@ceph.mgr.z.smithi133.stdout:Jun 22 17:21:26 smithi133 conmon[88969]: 2023-06-22T17:21:26.795+0000 7fe81b8051c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-06-22T17:21:27.214 INFO:teuthology.orchestra.run.smithi039.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-06-22T17:21:27.214 INFO:teuthology.orchestra.run.smithi039.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-06-22T17:21:27.214 INFO:teuthology.orchestra.run.smithi039.stderr: stderr: 10+0 records in 2023-06-22T17:21:27.214 INFO:teuthology.orchestra.run.smithi039.stderr:10+0 records out 2023-06-22T17:21:27.214 INFO:teuthology.orchestra.run.smithi039.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00729078 s, 1.4 GB/s 2023-06-22T17:21:27.214 INFO:teuthology.orchestra.run.smithi039.stderr:--> Zapping successful for: 2023-06-22T17:21:27.634 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph orch daemon add osd smithi039:vg_nvme/lv_4 2023-06-22T17:21:27.669 INFO:journalctl@ceph.mgr.z.smithi133.stdout:Jun 22 17:21:27 smithi133 conmon[88969]: 2023-06-22T17:21:27.409+0000 7fe81b8051c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-06-22T17:21:28.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:27 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:28.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:27 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:28.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:27 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:28.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:27 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:28.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:27 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:21:28.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:27 smithi039 ceph-mon[84805]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:28.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:27 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:28.229 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-22T17:21:28.288 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jun 22 17:21:28 smithi101 conmon[87163]: 2023-06-22T17:21:28.119+0000 7fb0e7afc1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-06-22T17:21:28.288 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:27 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:28.289 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:27 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:28.289 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:27 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:28.289 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:27 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:28.289 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:27 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:21:28.289 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:27 smithi101 ceph-mon[85507]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:28.289 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:27 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:28.419 INFO:journalctl@ceph.mgr.z.smithi133.stdout:Jun 22 17:21:27 smithi133 conmon[88969]: 2023-06-22T17:21:27.931+0000 7fe81b8051c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-06-22T17:21:28.419 INFO:journalctl@ceph.mgr.z.smithi133.stdout:Jun 22 17:21:28 smithi133 conmon[88969]: 2023-06-22T17:21:28.028+0000 7fe81b8051c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-06-22T17:21:28.420 INFO:journalctl@ceph.mgr.z.smithi133.stdout:Jun 22 17:21:28 smithi133 conmon[88969]: 2023-06-22T17:21:28.126+0000 7fe81b8051c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-06-22T17:21:28.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:27 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:28.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:27 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:28.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:27 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:28.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:27 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:28.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:27 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:21:28.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:27 smithi133 ceph-mon[86268]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:28.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:27 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:28.606 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jun 22 17:21:28 smithi101 conmon[87163]: 2023-06-22T17:21:28.287+0000 7fb0e7afc1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-06-22T17:21:28.606 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jun 22 17:21:28 smithi101 conmon[87163]: 2023-06-22T17:21:28.374+0000 7fb0e7afc1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-06-22T17:21:28.875 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jun 22 17:21:28 smithi101 conmon[87163]: 2023-06-22T17:21:28.606+0000 7fb0e7afc1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-06-22T17:21:28.875 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jun 22 17:21:28 smithi101 conmon[87163]: 2023-06-22T17:21:28.807+0000 7fb0e7afc1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-06-22T17:21:29.008 INFO:journalctl@ceph.mgr.z.smithi133.stdout:Jun 22 17:21:28 smithi133 conmon[88969]: 2023-06-22T17:21:28.739+0000 7fe81b8051c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-06-22T17:21:29.008 INFO:journalctl@ceph.mgr.z.smithi133.stdout:Jun 22 17:21:28 smithi133 conmon[88969]: 2023-06-22T17:21:28.913+0000 7fe81b8051c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-06-22T17:21:29.008 INFO:journalctl@ceph.mgr.z.smithi133.stdout:Jun 22 17:21:29 smithi133 conmon[88969]: 2023-06-22T17:21:29.006+0000 7fe81b8051c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-06-22T17:21:29.375 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jun 22 17:21:28 smithi101 conmon[87163]: 2023-06-22T17:21:28.969+0000 7fb0e7afc1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-06-22T17:21:29.375 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jun 22 17:21:29 smithi101 conmon[87163]: 2023-06-22T17:21:29.050+0000 7fb0e7afc1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-06-22T17:21:29.419 INFO:journalctl@ceph.mgr.z.smithi133.stdout:Jun 22 17:21:29 smithi133 conmon[88969]: 2023-06-22T17:21:29.135+0000 7fe81b8051c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-06-22T17:21:29.419 INFO:journalctl@ceph.mgr.z.smithi133.stdout:Jun 22 17:21:29 smithi133 conmon[88969]: 2023-06-22T17:21:29.226+0000 7fe81b8051c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-06-22T17:21:29.541 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.541+0000 7fd50d89e700 1 -- 172.21.15.39:0/3935345146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5080fbaf0 msgr2=0x7fd5080fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:21:29.542 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.541+0000 7fd50d89e700 1 --2- 172.21.15.39:0/3935345146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5080fbaf0 0x7fd5080fbf00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd4f80049c0 tx=0x7fd4f8009e30 comp rx=0 tx=0).stop 2023-06-22T17:21:29.542 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.541+0000 7fd50d89e700 1 -- 172.21.15.39:0/3935345146 shutdown_connections 2023-06-22T17:21:29.542 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.541+0000 7fd50d89e700 1 --2- 172.21.15.39:0/3935345146 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd5080fd750 0x7fd508103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:29.542 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.541+0000 7fd50d89e700 1 --2- 172.21.15.39:0/3935345146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5080fbaf0 0x7fd5080fbf00 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:29.542 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.541+0000 7fd50d89e700 1 --2- 172.21.15.39:0/3935345146 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd5080fcda0 0x7fd5080fd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:29.543 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.541+0000 7fd50d89e700 1 -- 172.21.15.39:0/3935345146 >> 172.21.15.39:0/3935345146 conn(0x7fd5080f7210 msgr2=0x7fd5080f9680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:21:29.543 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.541+0000 7fd50d89e700 1 -- 172.21.15.39:0/3935345146 shutdown_connections 2023-06-22T17:21:29.543 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.541+0000 7fd50d89e700 1 -- 172.21.15.39:0/3935345146 wait complete. 2023-06-22T17:21:29.545 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.541+0000 7fd50d89e700 1 Processor -- start 2023-06-22T17:21:29.545 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.542+0000 7fd50d89e700 1 -- start start 2023-06-22T17:21:29.545 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.542+0000 7fd50d89e700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd5080fbaf0 0x7fd50810c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:21:29.545 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.542+0000 7fd50d89e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5080fcda0 0x7fd50810cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:21:29.545 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.542+0000 7fd50d89e700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd5080fd750 0x7fd508113e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:21:29.545 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.542+0000 7fd50d89e700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd508114440 con 0x7fd5080fcda0 2023-06-22T17:21:29.545 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.542+0000 7fd50d89e700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5081145b0 con 0x7fd5080fbaf0 2023-06-22T17:21:29.546 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.542+0000 7fd50d89e700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd508114770 con 0x7fd5080fd750 2023-06-22T17:21:29.546 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.542+0000 7fd5067fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5080fcda0 0x7fd50810cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:21:29.546 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.542+0000 7fd5067fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5080fcda0 0x7fd50810cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57858/0 (socket says 172.21.15.39:57858) 2023-06-22T17:21:29.546 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.542+0000 7fd5067fc700 1 -- 172.21.15.39:0/4085168721 learned_addr learned my addr 172.21.15.39:0/4085168721 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:21:29.546 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.542+0000 7fd506ffd700 1 --2- 172.21.15.39:0/4085168721 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd5080fbaf0 0x7fd50810c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:21:29.546 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.542+0000 7fd5077fe700 1 --2- 172.21.15.39:0/4085168721 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd5080fd750 0x7fd508113e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:21:29.546 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.542+0000 7fd5067fc700 1 -- 172.21.15.39:0/4085168721 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd5080fd750 msgr2=0x7fd508113e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:21:29.547 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.542+0000 7fd5067fc700 1 --2- 172.21.15.39:0/4085168721 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd5080fd750 0x7fd508113e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:29.547 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.542+0000 7fd5067fc700 1 -- 172.21.15.39:0/4085168721 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd5080fbaf0 msgr2=0x7fd50810c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:21:29.547 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.542+0000 7fd5067fc700 1 --2- 172.21.15.39:0/4085168721 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd5080fbaf0 0x7fd50810c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:29.547 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.542+0000 7fd5067fc700 1 -- 172.21.15.39:0/4085168721 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f8005040 con 0x7fd5080fcda0 2023-06-22T17:21:29.547 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.542+0000 7fd5067fc700 1 --2- 172.21.15.39:0/4085168721 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5080fcda0 0x7fd50810cda0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc010dd0 tx=0x7fd4fc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:21:29.547 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.542+0000 7fd4ef7fe700 1 -- 172.21.15.39:0/4085168721 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4fc010ed0 con 0x7fd5080fcda0 2023-06-22T17:21:29.547 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.543+0000 7fd4ef7fe700 1 -- 172.21.15.39:0/4085168721 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7fd4fc007cc0 con 0x7fd5080fcda0 2023-06-22T17:21:29.547 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.543+0000 7fd50d89e700 1 -- 172.21.15.39:0/4085168721 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd508065430 con 0x7fd5080fcda0 2023-06-22T17:21:29.548 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.543+0000 7fd4ef7fe700 1 -- 172.21.15.39:0/4085168721 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4fc012430 con 0x7fd5080fcda0 2023-06-22T17:21:29.548 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.543+0000 7fd50d89e700 1 -- 172.21.15.39:0/4085168721 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5080659f0 con 0x7fd5080fcda0 2023-06-22T17:21:29.548 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.543+0000 7fd4ef7fe700 1 -- 172.21.15.39:0/4085168721 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) v1 ==== 43275+0+0 (secure 0 0 0) 0x7fd4fc018030 con 0x7fd5080fcda0 2023-06-22T17:21:29.548 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.543+0000 7fd50d89e700 1 -- 172.21.15.39:0/4085168721 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5080663e0 con 0x7fd5080fcda0 2023-06-22T17:21:29.548 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.544+0000 7fd4ef7fe700 1 --2- 172.21.15.39:0/4085168721 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fd4f0036890 0x7fd4f0038d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:21:29.548 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.544+0000 7fd4ef7fe700 1 -- 172.21.15.39:0/4085168721 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fd4fc016070 con 0x7fd5080fcda0 2023-06-22T17:21:29.548 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.544+0000 7fd506ffd700 1 --2- 172.21.15.39:0/4085168721 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fd4f0036890 0x7fd4f0038d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:21:29.548 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.544+0000 7fd506ffd700 1 --2- 172.21.15.39:0/4085168721 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fd4f0036890 0x7fd4f0038d40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd4f800b3b0 tx=0x7fd4f8009e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:21:29.549 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.548+0000 7fd4ef7fe700 1 -- 172.21.15.39:0/4085168721 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fd4fc0224e0 con 0x7fd5080fcda0 2023-06-22T17:21:29.762 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:29.761+0000 7fd50d89e700 1 -- 172.21.15.39:0/4085168721 --> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fd508102740 con 0x7fd4f0036890 2023-06-22T17:21:29.875 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jun 22 17:21:29 smithi101 conmon[87163]: 2023-06-22T17:21:29.636+0000 7fb0e7afc1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-06-22T17:21:29.875 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:29 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:29.875 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:29 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:29.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:29 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:29.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:29 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:29.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:29 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:29.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:29 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:21:29.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:29 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:29.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:29 smithi101 ceph-mon[85507]: Reconfiguring mgr.x (unknown last config time)... 2023-06-22T17:21:29.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:29 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-06-22T17:21:29.878 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:29 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-06-22T17:21:29.878 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:29 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:29.878 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:29 smithi101 ceph-mon[85507]: Reconfiguring daemon mgr.x on smithi039 2023-06-22T17:21:29.878 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:29 smithi101 ceph-mon[85507]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:29.879 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:29 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:29.879 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:29 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:29.879 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:29 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:29.879 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:29 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:29.879 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:29 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:29.879 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:29 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:21:29.879 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:29 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:29.880 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:29 smithi039 ceph-mon[84805]: Reconfiguring mgr.x (unknown last config time)... 2023-06-22T17:21:29.880 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:29 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-06-22T17:21:29.880 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:29 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-06-22T17:21:29.880 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:29 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:29.880 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:29 smithi039 ceph-mon[84805]: Reconfiguring daemon mgr.x on smithi039 2023-06-22T17:21:29.880 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:29 smithi039 ceph-mon[84805]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:29.919 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:29 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:29.919 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:29 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:29.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:29 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:29.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:29 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:29.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:29 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:29.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:29 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:21:29.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:29 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:29.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:29 smithi133 ceph-mon[86268]: Reconfiguring mgr.x (unknown last config time)... 2023-06-22T17:21:29.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:29 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-06-22T17:21:29.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:29 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-06-22T17:21:29.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:29 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:29.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:29 smithi133 ceph-mon[86268]: Reconfiguring daemon mgr.x on smithi039 2023-06-22T17:21:29.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:29 smithi133 ceph-mon[86268]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:30.375 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jun 22 17:21:30 smithi101 conmon[87163]: 2023-06-22T17:21:30.120+0000 7fb0e7afc1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-06-22T17:21:30.375 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jun 22 17:21:30 smithi101 conmon[87163]: 2023-06-22T17:21:30.212+0000 7fb0e7afc1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-06-22T17:21:30.375 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jun 22 17:21:30 smithi101 conmon[87163]: 2023-06-22T17:21:30.305+0000 7fb0e7afc1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-06-22T17:21:30.419 INFO:journalctl@ceph.mgr.z.smithi133.stdout:Jun 22 17:21:30 smithi133 conmon[88969]: 2023-06-22T17:21:30.101+0000 7fe81b8051c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-06-22T17:21:30.419 INFO:journalctl@ceph.mgr.z.smithi133.stdout:Jun 22 17:21:30 smithi133 conmon[88969]: 2023-06-22T17:21:30.188+0000 7fe81b8051c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-06-22T17:21:30.420 INFO:journalctl@ceph.mgr.z.smithi133.stdout:Jun 22 17:21:30 smithi133 conmon[88969]: 2023-06-22T17:21:30.276+0000 7fe81b8051c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-06-22T17:21:30.888 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:30 smithi101 ceph-mon[85507]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:21:30.888 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:30 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:21:30.889 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:30 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:21:30.889 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:30 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:30.889 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:30 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:30.889 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:30 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:30.889 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:30 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:21:30.915 INFO:journalctl@ceph.mgr.z.smithi133.stdout:Jun 22 17:21:30 smithi133 conmon[88969]: 2023-06-22T17:21:30.423+0000 7fe81b8051c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-06-22T17:21:30.916 INFO:journalctl@ceph.mgr.z.smithi133.stdout:Jun 22 17:21:30 smithi133 conmon[88969]: 2023-06-22T17:21:30.522+0000 7fe81b8051c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-06-22T17:21:30.916 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:30 smithi133 ceph-mon[86268]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:21:30.916 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:30 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:21:30.916 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:30 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:21:30.916 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:30 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:30.917 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:30 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:30.917 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:30 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:30.917 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:30 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:21:30.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:30 smithi039 ceph-mon[84805]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:21:30.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:30 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:21:30.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:30 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:21:30.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:30 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:30.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:30 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:30.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:30 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:30.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:30 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:21:31.150 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jun 22 17:21:30 smithi101 conmon[87163]: 2023-06-22T17:21:30.888+0000 7fb0e7afc1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-06-22T17:21:31.150 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jun 22 17:21:31 smithi101 conmon[87163]: 2023-06-22T17:21:31.059+0000 7fb0e7afc1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-06-22T17:21:31.169 INFO:journalctl@ceph.mgr.z.smithi133.stdout:Jun 22 17:21:30 smithi133 conmon[88969]: 2023-06-22T17:21:30.914+0000 7fe81b8051c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-06-22T17:21:31.169 INFO:journalctl@ceph.mgr.z.smithi133.stdout:Jun 22 17:21:31 smithi133 conmon[88969]: 2023-06-22T17:21:31.006+0000 7fe81b8051c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-06-22T17:21:31.169 INFO:journalctl@ceph.mgr.z.smithi133.stdout:Jun 22 17:21:31 smithi133 conmon[88969]: 2023-06-22T17:21:31.095+0000 7fe81b8051c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-06-22T17:21:31.625 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jun 22 17:21:31 smithi101 conmon[87163]: 2023-06-22T17:21:31.150+0000 7fb0e7afc1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-06-22T17:21:31.625 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jun 22 17:21:31 smithi101 conmon[87163]: 2023-06-22T17:21:31.277+0000 7fb0e7afc1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-06-22T17:21:31.625 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jun 22 17:21:31 smithi101 conmon[87163]: 2023-06-22T17:21:31.367+0000 7fb0e7afc1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-06-22T17:21:31.629 INFO:journalctl@ceph.mgr.z.smithi133.stdout:Jun 22 17:21:31 smithi133 conmon[88969]: 2023-06-22T17:21:31.183+0000 7fe81b8051c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-06-22T17:21:31.886 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:31.885+0000 7fd4ef7fe700 1 -- 172.21.15.39:0/4085168721 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mgrmap(e 14) v1 ==== 86210+0+0 (secure 0 0 0) 0x7fd4fc0113c0 con 0x7fd5080fcda0 2023-06-22T17:21:31.919 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:31 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:31.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:31 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:21:31.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:31 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:31.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:31 smithi133 ceph-mon[86268]: Standby manager daemon z started 2023-06-22T17:21:31.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:31 smithi133 ceph-mon[86268]: from='mgr.? 172.21.15.133:0/2848148404' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-06-22T17:21:31.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:31 smithi133 ceph-mon[86268]: from='mgr.? 172.21.15.133:0/2848148404' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-06-22T17:21:31.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:31 smithi133 ceph-mon[86268]: from='mgr.? 172.21.15.133:0/2848148404' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-06-22T17:21:31.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:31 smithi133 ceph-mon[86268]: from='mgr.? 172.21.15.133:0/2848148404' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-06-22T17:21:31.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:31 smithi133 ceph-mon[86268]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:31.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:31 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:31.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:31 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:21:31.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:31 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:31.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:31 smithi039 ceph-mon[84805]: Standby manager daemon z started 2023-06-22T17:21:31.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:31 smithi039 ceph-mon[84805]: from='mgr.? 172.21.15.133:0/2848148404' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-06-22T17:21:31.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:31 smithi039 ceph-mon[84805]: from='mgr.? 172.21.15.133:0/2848148404' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-06-22T17:21:31.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:31 smithi039 ceph-mon[84805]: from='mgr.? 172.21.15.133:0/2848148404' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-06-22T17:21:31.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:31 smithi039 ceph-mon[84805]: from='mgr.? 172.21.15.133:0/2848148404' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-06-22T17:21:31.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:31 smithi039 ceph-mon[84805]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:32.125 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:31 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:32.125 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:31 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:21:32.125 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:31 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:32.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:31 smithi101 ceph-mon[85507]: Standby manager daemon z started 2023-06-22T17:21:32.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:31 smithi101 ceph-mon[85507]: from='mgr.? 172.21.15.133:0/2848148404' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-06-22T17:21:32.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:31 smithi101 ceph-mon[85507]: from='mgr.? 172.21.15.133:0/2848148404' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-06-22T17:21:32.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:31 smithi101 ceph-mon[85507]: from='mgr.? 172.21.15.133:0/2848148404' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-06-22T17:21:32.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:31 smithi101 ceph-mon[85507]: from='mgr.? 172.21.15.133:0/2848148404' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-06-22T17:21:32.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:31 smithi101 ceph-mon[85507]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:32.557 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jun 22 17:21:32 smithi101 conmon[87163]: 2023-06-22T17:21:32.235+0000 7fb0e7afc1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-06-22T17:21:32.558 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jun 22 17:21:32 smithi101 conmon[87163]: 2023-06-22T17:21:32.322+0000 7fb0e7afc1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-06-22T17:21:32.558 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jun 22 17:21:32 smithi101 conmon[87163]: 2023-06-22T17:21:32.411+0000 7fb0e7afc1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-06-22T17:21:32.875 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jun 22 17:21:32 smithi101 conmon[87163]: 2023-06-22T17:21:32.557+0000 7fb0e7afc1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-06-22T17:21:32.875 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jun 22 17:21:32 smithi101 conmon[87163]: 2023-06-22T17:21:32.653+0000 7fb0e7afc1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-06-22T17:21:33.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:32 smithi133 ceph-mon[86268]: mgrmap e14: x(active, since 64s), standbys: z 2023-06-22T17:21:33.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:32 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-06-22T17:21:33.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:32 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/3366482975' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb8f5cba-9266-47ec-8af1-8e9edd6ec96a"}]: dispatch 2023-06-22T17:21:33.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:32 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/3366482975' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb8f5cba-9266-47ec-8af1-8e9edd6ec96a"}]': finished 2023-06-22T17:21:33.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:32 smithi133 ceph-mon[86268]: osdmap e5: 1 total, 0 up, 1 in 2023-06-22T17:21:33.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:32 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-22T17:21:33.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:32 smithi039 ceph-mon[84805]: mgrmap e14: x(active, since 64s), standbys: z 2023-06-22T17:21:33.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:32 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-06-22T17:21:33.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:32 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/3366482975' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb8f5cba-9266-47ec-8af1-8e9edd6ec96a"}]: dispatch 2023-06-22T17:21:33.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:32 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/3366482975' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb8f5cba-9266-47ec-8af1-8e9edd6ec96a"}]': finished 2023-06-22T17:21:33.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:32 smithi039 ceph-mon[84805]: osdmap e5: 1 total, 0 up, 1 in 2023-06-22T17:21:33.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:32 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-22T17:21:33.218 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jun 22 17:21:33 smithi101 conmon[87163]: 2023-06-22T17:21:33.039+0000 7fb0e7afc1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-06-22T17:21:33.218 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jun 22 17:21:33 smithi101 conmon[87163]: 2023-06-22T17:21:33.129+0000 7fb0e7afc1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-06-22T17:21:33.218 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:32 smithi101 ceph-mon[85507]: mgrmap e14: x(active, since 64s), standbys: z 2023-06-22T17:21:33.218 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:32 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-06-22T17:21:33.219 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:32 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/3366482975' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb8f5cba-9266-47ec-8af1-8e9edd6ec96a"}]: dispatch 2023-06-22T17:21:33.219 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:32 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/3366482975' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb8f5cba-9266-47ec-8af1-8e9edd6ec96a"}]': finished 2023-06-22T17:21:33.219 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:32 smithi101 ceph-mon[85507]: osdmap e5: 1 total, 0 up, 1 in 2023-06-22T17:21:33.219 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:32 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-22T17:21:33.625 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jun 22 17:21:33 smithi101 conmon[87163]: 2023-06-22T17:21:33.217+0000 7fb0e7afc1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-06-22T17:21:33.625 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jun 22 17:21:33 smithi101 conmon[87163]: 2023-06-22T17:21:33.303+0000 7fb0e7afc1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-06-22T17:21:34.298 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:34.297+0000 7fd4ef7fe700 1 -- 172.21.15.39:0/4085168721 <== mon.0 v2:172.21.15.39:3300/0 8 ==== mgrmap(e 15) v1 ==== 129145+0+0 (secure 0 0 0) 0x7fd4fc00eaa0 con 0x7fd5080fcda0 2023-06-22T17:21:34.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:34 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/3075175469' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:21:34.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:34 smithi101 ceph-mon[85507]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:34.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:34 smithi101 ceph-mon[85507]: Standby manager daemon y started 2023-06-22T17:21:34.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:34 smithi101 ceph-mon[85507]: from='mgr.? 172.21.15.101:0/2395906255' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-06-22T17:21:34.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:34 smithi101 ceph-mon[85507]: from='mgr.? 172.21.15.101:0/2395906255' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-06-22T17:21:34.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:34 smithi101 ceph-mon[85507]: from='mgr.? 172.21.15.101:0/2395906255' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-06-22T17:21:34.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:34 smithi101 ceph-mon[85507]: from='mgr.? 172.21.15.101:0/2395906255' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-06-22T17:21:34.669 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:34 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/3075175469' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:21:34.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:34 smithi133 ceph-mon[86268]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:34.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:34 smithi133 ceph-mon[86268]: Standby manager daemon y started 2023-06-22T17:21:34.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:34 smithi133 ceph-mon[86268]: from='mgr.? 172.21.15.101:0/2395906255' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-06-22T17:21:34.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:34 smithi133 ceph-mon[86268]: from='mgr.? 172.21.15.101:0/2395906255' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-06-22T17:21:34.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:34 smithi133 ceph-mon[86268]: from='mgr.? 172.21.15.101:0/2395906255' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-06-22T17:21:34.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:34 smithi133 ceph-mon[86268]: from='mgr.? 172.21.15.101:0/2395906255' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-06-22T17:21:34.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:34 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/3075175469' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:21:34.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:34 smithi039 ceph-mon[84805]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:34.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:34 smithi039 ceph-mon[84805]: Standby manager daemon y started 2023-06-22T17:21:34.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:34 smithi039 ceph-mon[84805]: from='mgr.? 172.21.15.101:0/2395906255' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-06-22T17:21:34.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:34 smithi039 ceph-mon[84805]: from='mgr.? 172.21.15.101:0/2395906255' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-06-22T17:21:34.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:34 smithi039 ceph-mon[84805]: from='mgr.? 172.21.15.101:0/2395906255' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-06-22T17:21:34.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:34 smithi039 ceph-mon[84805]: from='mgr.? 172.21.15.101:0/2395906255' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-06-22T17:21:35.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:35 smithi101 ceph-mon[85507]: mgrmap e15: x(active, since 67s), standbys: y, z 2023-06-22T17:21:35.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:35 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-06-22T17:21:35.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:35 smithi133 ceph-mon[86268]: mgrmap e15: x(active, since 67s), standbys: y, z 2023-06-22T17:21:35.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:35 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-06-22T17:21:35.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:35 smithi039 ceph-mon[84805]: mgrmap e15: x(active, since 67s), standbys: y, z 2023-06-22T17:21:35.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:35 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-06-22T17:21:36.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:36 smithi101 ceph-mon[85507]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:36.669 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:36 smithi133 ceph-mon[86268]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:36.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:36 smithi039 ceph-mon[84805]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:38.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:38 smithi101 ceph-mon[85507]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:38.669 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:38 smithi133 ceph-mon[86268]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:38.678 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:38 smithi039 ceph-mon[84805]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:40.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:40 smithi039 ceph-mon[84805]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:40.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:40 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-06-22T17:21:40.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:40 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:40.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:40 smithi101 ceph-mon[85507]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:40.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:40 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-06-22T17:21:40.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:40 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:40.669 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:40 smithi133 ceph-mon[86268]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:40.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:40 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-06-22T17:21:40.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:40 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:41.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:41 smithi101 ceph-mon[85507]: Deploying daemon osd.0 on smithi039 2023-06-22T17:21:41.667 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:41 smithi039 ceph-mon[84805]: Deploying daemon osd.0 on smithi039 2023-06-22T17:21:41.669 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:41 smithi133 ceph-mon[86268]: Deploying daemon osd.0 on smithi039 2023-06-22T17:21:42.619 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:42 smithi039 ceph-mon[84805]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:42.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:42 smithi101 ceph-mon[85507]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:42.669 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:42 smithi133 ceph-mon[86268]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:44.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:44 smithi101 ceph-mon[85507]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:44.648 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:44 smithi039 ceph-mon[84805]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:44.669 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:44 smithi133 ceph-mon[86268]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:45.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:45 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:45.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:45 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:45.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:45 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:21:45.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:45 smithi039 ceph-mon[84805]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:46.125 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:45 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:46.125 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:45 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:46.125 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:45 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:21:46.125 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:45 smithi101 ceph-mon[85507]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:46.169 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:45 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:46.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:45 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:46.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:45 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:21:46.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:45 smithi133 ceph-mon[86268]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:47.717 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:47.715+0000 7fd4ef7fe700 1 -- 172.21.15.39:0/4085168721 <== mgr.14150 v2:172.21.15.39:6800/886078282 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd508102740 con 0x7fd4f0036890 2023-06-22T17:21:47.717 INFO:teuthology.orchestra.run.smithi039.stdout:Created osd(s) 0 on host 'smithi039' 2023-06-22T17:21:47.719 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:47.718+0000 7fd50d89e700 1 -- 172.21.15.39:0/4085168721 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fd4f0036890 msgr2=0x7fd4f0038d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:21:47.719 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:47.718+0000 7fd50d89e700 1 --2- 172.21.15.39:0/4085168721 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fd4f0036890 0x7fd4f0038d40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd4f800b3b0 tx=0x7fd4f8009e30 comp rx=0 tx=0).stop 2023-06-22T17:21:47.719 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:47.718+0000 7fd50d89e700 1 -- 172.21.15.39:0/4085168721 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5080fcda0 msgr2=0x7fd50810cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:21:47.719 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:47.718+0000 7fd50d89e700 1 --2- 172.21.15.39:0/4085168721 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5080fcda0 0x7fd50810cda0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc010dd0 tx=0x7fd4fc00b040 comp rx=0 tx=0).stop 2023-06-22T17:21:47.719 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:47.718+0000 7fd50d89e700 1 -- 172.21.15.39:0/4085168721 shutdown_connections 2023-06-22T17:21:47.720 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:47.718+0000 7fd50d89e700 1 --2- 172.21.15.39:0/4085168721 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fd4f0036890 0x7fd4f0038d40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:47.720 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:47.718+0000 7fd50d89e700 1 --2- 172.21.15.39:0/4085168721 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd5080fd750 0x7fd508113e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:47.720 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:47.718+0000 7fd50d89e700 1 --2- 172.21.15.39:0/4085168721 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd5080fbaf0 0x7fd50810c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:47.720 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:47.718+0000 7fd50d89e700 1 --2- 172.21.15.39:0/4085168721 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5080fcda0 0x7fd50810cda0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:47.720 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:47.718+0000 7fd50d89e700 1 -- 172.21.15.39:0/4085168721 >> 172.21.15.39:0/4085168721 conn(0x7fd5080f7210 msgr2=0x7fd5080f93c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:21:47.720 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:47.718+0000 7fd50d89e700 1 -- 172.21.15.39:0/4085168721 shutdown_connections 2023-06-22T17:21:47.720 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:21:47.718+0000 7fd50d89e700 1 -- 172.21.15.39:0/4085168721 wait complete. 2023-06-22T17:21:47.875 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:47 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:47.875 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:47 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:47.875 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:47 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:47.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:47 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:21:47.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:47 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:47.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:47 smithi101 ceph-mon[85507]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:47.919 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:47 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:47.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:47 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:47.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:47 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:47.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:47 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:21:47.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:47 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:47.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:47 smithi133 ceph-mon[86268]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:47.928 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:47 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:47.928 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:47 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:47.929 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:47 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:47.929 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:47 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:21:47.929 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:47 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:47.929 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:47 smithi039 ceph-mon[84805]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:48.212 DEBUG:teuthology.orchestra.run.smithi039:osd.0> sudo journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.0.service 2023-06-22T17:21:48.214 INFO:tasks.cephadm:Deploying osd.1 on smithi101 with /dev/vg_nvme/lv_4... 2023-06-22T17:21:48.215 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-06-22T17:21:48.246 INFO:journalctl@ceph.osd.0.smithi039.stdout:-- Logs begin at Thu 2023-06-22 17:07:05 UTC. -- 2023-06-22T17:21:48.792 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.b/config 2023-06-22T17:21:48.849 INFO:journalctl@ceph.osd.0.smithi039.stdout:Jun 22 17:21:48 smithi039 conmon[101634]: 2023-06-22T17:21:48.553+0000 7f4c3380d540 -1 osd.0 0 log_to_monitors true 2023-06-22T17:21:48.850 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:48 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:21:48.850 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:48 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:48.850 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:48 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:48.851 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:48 smithi039 ceph-mon[84805]: from='osd.0 [v2:172.21.15.39:6802/453150307,v1:172.21.15.39:6803/453150307]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-06-22T17:21:48.851 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:48 smithi039 ceph-mon[84805]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-06-22T17:21:48.875 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:48 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:21:48.875 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:48 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:48.875 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:48 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:48.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:48 smithi101 ceph-mon[85507]: from='osd.0 [v2:172.21.15.39:6802/453150307,v1:172.21.15.39:6803/453150307]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-06-22T17:21:48.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:48 smithi101 ceph-mon[85507]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-06-22T17:21:48.919 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:48 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:21:48.919 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:48 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:48.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:48 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:48.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:48 smithi133 ceph-mon[86268]: from='osd.0 [v2:172.21.15.39:6802/453150307,v1:172.21.15.39:6803/453150307]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-06-22T17:21:48.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:48 smithi133 ceph-mon[86268]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-06-22T17:21:49.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:49 smithi039 ceph-mon[84805]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-06-22T17:21:49.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:49 smithi039 ceph-mon[84805]: osdmap e6: 1 total, 0 up, 1 in 2023-06-22T17:21:49.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:49 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-22T17:21:49.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:49 smithi039 ceph-mon[84805]: from='osd.0 [v2:172.21.15.39:6802/453150307,v1:172.21.15.39:6803/453150307]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-06-22T17:21:49.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:49 smithi039 ceph-mon[84805]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-06-22T17:21:49.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:49 smithi039 ceph-mon[84805]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:49.934 INFO:journalctl@ceph.osd.0.smithi039.stdout:Jun 22 17:21:49 smithi039 conmon[101634]: 2023-06-22T17:21:49.717+0000 7f4c29ff2700 -1 osd.0 0 waiting for initial osdmap 2023-06-22T17:21:49.934 INFO:journalctl@ceph.osd.0.smithi039.stdout:Jun 22 17:21:49 smithi039 conmon[101634]: 2023-06-22T17:21:49.720+0000 7f4c26df1700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-22T17:21:50.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:49 smithi101 ceph-mon[85507]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-06-22T17:21:50.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:49 smithi101 ceph-mon[85507]: osdmap e6: 1 total, 0 up, 1 in 2023-06-22T17:21:50.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:49 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-22T17:21:50.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:49 smithi101 ceph-mon[85507]: from='osd.0 [v2:172.21.15.39:6802/453150307,v1:172.21.15.39:6803/453150307]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-06-22T17:21:50.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:49 smithi101 ceph-mon[85507]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-06-22T17:21:50.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:49 smithi101 ceph-mon[85507]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:50.169 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:49 smithi133 ceph-mon[86268]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-06-22T17:21:50.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:49 smithi133 ceph-mon[86268]: osdmap e6: 1 total, 0 up, 1 in 2023-06-22T17:21:50.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:49 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-22T17:21:50.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:49 smithi133 ceph-mon[86268]: from='osd.0 [v2:172.21.15.39:6802/453150307,v1:172.21.15.39:6803/453150307]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-06-22T17:21:50.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:49 smithi133 ceph-mon[86268]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-06-22T17:21:50.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:49 smithi133 ceph-mon[86268]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:50.405 INFO:teuthology.orchestra.run.smithi101.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-06-22T17:21:50.406 INFO:teuthology.orchestra.run.smithi101.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-06-22T17:21:50.406 INFO:teuthology.orchestra.run.smithi101.stderr: stderr: 10+0 records in 2023-06-22T17:21:50.406 INFO:teuthology.orchestra.run.smithi101.stderr:10+0 records out 2023-06-22T17:21:50.406 INFO:teuthology.orchestra.run.smithi101.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112648 s, 931 MB/s 2023-06-22T17:21:50.406 INFO:teuthology.orchestra.run.smithi101.stderr:--> Zapping successful for: 2023-06-22T17:21:50.824 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph orch daemon add osd smithi101:vg_nvme/lv_4 2023-06-22T17:21:50.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:50 smithi039 ceph-mon[84805]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2023-06-22T17:21:50.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:50 smithi039 ceph-mon[84805]: osdmap e7: 1 total, 0 up, 1 in 2023-06-22T17:21:50.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:50 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-22T17:21:50.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:50 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-22T17:21:50.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:50 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:50.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:50 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:50.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:50 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:21:50.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:50 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:50.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:50 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:50.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:50 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:21:50.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:50 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:51.125 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:50 smithi101 ceph-mon[85507]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2023-06-22T17:21:51.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:50 smithi101 ceph-mon[85507]: osdmap e7: 1 total, 0 up, 1 in 2023-06-22T17:21:51.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:50 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-22T17:21:51.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:50 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-22T17:21:51.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:50 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:51.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:50 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:51.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:50 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:21:51.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:50 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:51.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:50 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:51.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:50 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:21:51.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:50 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:51.169 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:50 smithi133 ceph-mon[86268]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2023-06-22T17:21:51.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:50 smithi133 ceph-mon[86268]: osdmap e7: 1 total, 0 up, 1 in 2023-06-22T17:21:51.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:50 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-22T17:21:51.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:50 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-22T17:21:51.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:50 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:51.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:50 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:51.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:50 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:21:51.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:50 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:51.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:50 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:51.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:50 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:21:51.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:50 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:21:51.401 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.b/config 2023-06-22T17:21:51.875 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:51 smithi101 ceph-mon[85507]: purged_snaps scrub starts 2023-06-22T17:21:51.875 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:51 smithi101 ceph-mon[85507]: purged_snaps scrub ok 2023-06-22T17:21:51.875 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:51 smithi101 ceph-mon[85507]: Detected new or changed devices on smithi039 2023-06-22T17:21:51.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:51 smithi101 ceph-mon[85507]: Adjusting osd_memory_target on smithi039 to 17168M 2023-06-22T17:21:51.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:51 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-22T17:21:51.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:51 smithi101 ceph-mon[85507]: osd.0 [v2:172.21.15.39:6802/453150307,v1:172.21.15.39:6803/453150307] boot 2023-06-22T17:21:51.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:51 smithi101 ceph-mon[85507]: osdmap e8: 1 total, 1 up, 1 in 2023-06-22T17:21:51.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:51 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-22T17:21:51.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:51 smithi101 ceph-mon[85507]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:52.169 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:51 smithi133 ceph-mon[86268]: purged_snaps scrub starts 2023-06-22T17:21:52.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:51 smithi133 ceph-mon[86268]: purged_snaps scrub ok 2023-06-22T17:21:52.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:51 smithi133 ceph-mon[86268]: Detected new or changed devices on smithi039 2023-06-22T17:21:52.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:51 smithi133 ceph-mon[86268]: Adjusting osd_memory_target on smithi039 to 17168M 2023-06-22T17:21:52.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:51 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-22T17:21:52.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:51 smithi133 ceph-mon[86268]: osd.0 [v2:172.21.15.39:6802/453150307,v1:172.21.15.39:6803/453150307] boot 2023-06-22T17:21:52.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:51 smithi133 ceph-mon[86268]: osdmap e8: 1 total, 1 up, 1 in 2023-06-22T17:21:52.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:51 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-22T17:21:52.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:51 smithi133 ceph-mon[86268]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:52.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:51 smithi039 ceph-mon[84805]: purged_snaps scrub starts 2023-06-22T17:21:52.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:51 smithi039 ceph-mon[84805]: purged_snaps scrub ok 2023-06-22T17:21:52.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:51 smithi039 ceph-mon[84805]: Detected new or changed devices on smithi039 2023-06-22T17:21:52.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:51 smithi039 ceph-mon[84805]: Adjusting osd_memory_target on smithi039 to 17168M 2023-06-22T17:21:52.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:51 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-22T17:21:52.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:51 smithi039 ceph-mon[84805]: osd.0 [v2:172.21.15.39:6802/453150307,v1:172.21.15.39:6803/453150307] boot 2023-06-22T17:21:52.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:51 smithi039 ceph-mon[84805]: osdmap e8: 1 total, 1 up, 1 in 2023-06-22T17:21:52.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:51 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-22T17:21:52.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:51 smithi039 ceph-mon[84805]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-22T17:21:52.593 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.590+0000 7f9457a39700 1 -- 172.21.15.101:0/952363345 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f94500fe820 msgr2=0x7f94501050e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:21:52.593 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.590+0000 7f9457a39700 1 --2- 172.21.15.101:0/952363345 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f94500fe820 0x7f94501050e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9448009d60 tx=0x7f9448004d10 comp rx=0 tx=0).stop 2023-06-22T17:21:52.593 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.590+0000 7f9457a39700 1 -- 172.21.15.101:0/952363345 shutdown_connections 2023-06-22T17:21:52.594 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.590+0000 7f9457a39700 1 --2- 172.21.15.101:0/952363345 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f94500fe820 0x7f94501050e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:52.594 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.590+0000 7f9457a39700 1 --2- 172.21.15.101:0/952363345 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f94500fcc80 0x7f94500fd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:52.594 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.590+0000 7f9457a39700 1 --2- 172.21.15.101:0/952363345 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94500fde90 0x7f94500fe2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:52.594 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.590+0000 7f9457a39700 1 -- 172.21.15.101:0/952363345 >> 172.21.15.101:0/952363345 conn(0x7f94500f83a0 msgr2=0x7f94500fa810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:21:52.594 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.590+0000 7f9457a39700 1 -- 172.21.15.101:0/952363345 shutdown_connections 2023-06-22T17:21:52.594 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.591+0000 7f9457a39700 1 -- 172.21.15.101:0/952363345 wait complete. 2023-06-22T17:21:52.594 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.591+0000 7f9457a39700 1 Processor -- start 2023-06-22T17:21:52.594 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.591+0000 7f9457a39700 1 -- start start 2023-06-22T17:21:52.595 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.591+0000 7f9457a39700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f94500fcc80 0x7f9450113f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:21:52.595 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.591+0000 7f9457a39700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f94500fde90 0x7f94501144c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:21:52.595 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.591+0000 7f9457a39700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94500fe820 0x7f945011b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:21:52.595 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.591+0000 7f9457a39700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945011bb60 con 0x7f94500fe820 2023-06-22T17:21:52.595 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.591+0000 7f9457a39700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945011bcd0 con 0x7f94500fde90 2023-06-22T17:21:52.595 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.591+0000 7f9457a39700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945011be90 con 0x7f94500fcc80 2023-06-22T17:21:52.595 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.591+0000 7f9454fd4700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f94500fde90 0x7f94501144c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:21:52.595 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.591+0000 7f9454fd4700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f94500fde90 0x7f94501144c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:50124/0 (socket says 172.21.15.101:50124) 2023-06-22T17:21:52.596 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.591+0000 7f9454fd4700 1 -- 172.21.15.101:0/1265751273 learned_addr learned my addr 172.21.15.101:0/1265751273 (peer_addr_for_me v2:172.21.15.101:0/0) 2023-06-22T17:21:52.596 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.592+0000 7f9455fd6700 1 --2- 172.21.15.101:0/1265751273 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94500fe820 0x7f945011b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:21:52.596 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.592+0000 7f94557d5700 1 --2- 172.21.15.101:0/1265751273 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f94500fcc80 0x7f9450113f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:21:52.596 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.592+0000 7f9454fd4700 1 -- 172.21.15.101:0/1265751273 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f94500fcc80 msgr2=0x7f9450113f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:21:52.596 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.592+0000 7f9454fd4700 1 --2- 172.21.15.101:0/1265751273 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f94500fcc80 0x7f9450113f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:52.596 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.592+0000 7f9454fd4700 1 -- 172.21.15.101:0/1265751273 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94500fe820 msgr2=0x7f945011b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:21:52.597 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.592+0000 7f9454fd4700 1 --2- 172.21.15.101:0/1265751273 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94500fe820 0x7f945011b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:21:52.597 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.592+0000 7f9454fd4700 1 -- 172.21.15.101:0/1265751273 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9448009940 con 0x7f94500fde90 2023-06-22T17:21:52.597 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.592+0000 7f9454fd4700 1 --2- 172.21.15.101:0/1265751273 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f94500fde90 0x7f94501144c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f944000b810 tx=0x7f9440009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:21:52.597 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.592+0000 7f9445ffb700 1 -- 172.21.15.101:0/1265751273 <== mon.2 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944000f510 con 0x7f94500fde90 2023-06-22T17:21:52.597 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.592+0000 7f9445ffb700 1 -- 172.21.15.101:0/1265751273 <== mon.2 v2:172.21.15.101:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f944000fb50 con 0x7f94500fde90 2023-06-22T17:21:52.597 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.592+0000 7f9457a39700 1 -- 172.21.15.101:0/1265751273 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f945011c170 con 0x7f94500fde90 2023-06-22T17:21:52.597 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.592+0000 7f9445ffb700 1 -- 172.21.15.101:0/1265751273 <== mon.2 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9440011cd0 con 0x7f94500fde90 2023-06-22T17:21:52.598 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.592+0000 7f9457a39700 1 -- 172.21.15.101:0/1265751273 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f945011c6b0 con 0x7f94500fde90 2023-06-22T17:21:52.598 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.593+0000 7f9457a39700 1 -- 172.21.15.101:0/1265751273 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9450102050 con 0x7f94500fde90 2023-06-22T17:21:52.599 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.594+0000 7f9445ffb700 1 -- 172.21.15.101:0/1265751273 <== mon.2 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 15) v1 ==== 129145+0+0 (secure 0 0 0) 0x7f944000fcc0 con 0x7f94500fde90 2023-06-22T17:21:52.599 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.595+0000 7f9445ffb700 1 --2- 172.21.15.101:0/1265751273 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f943c09b670 0x7f943c09db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:21:52.599 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.595+0000 7f9445ffb700 1 -- 172.21.15.101:0/1265751273 <== mon.2 v2:172.21.15.101:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1763+0+0 (secure 0 0 0) 0x7f94400c2f30 con 0x7f94500fde90 2023-06-22T17:21:52.599 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.595+0000 7f94557d5700 1 --2- 172.21.15.101:0/1265751273 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f943c09b670 0x7f943c09db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:21:52.599 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.595+0000 7f94557d5700 1 --2- 172.21.15.101:0/1265751273 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f943c09b670 0x7f943c09db20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f944c006fd0 tx=0x7f944c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:21:52.600 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.599+0000 7f9445ffb700 1 -- 172.21.15.101:0/1265751273 <== mon.2 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f9440090200 con 0x7f94500fde90 2023-06-22T17:21:52.792 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:21:52.791+0000 7f9457a39700 1 -- 172.21.15.101:0/1265751273 --> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f945002b130 con 0x7f943c09b670 2023-06-22T17:21:53.125 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:52 smithi101 ceph-mon[85507]: osdmap e9: 1 total, 1 up, 1 in 2023-06-22T17:21:53.169 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:52 smithi133 ceph-mon[86268]: osdmap e9: 1 total, 1 up, 1 in 2023-06-22T17:21:53.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:52 smithi039 ceph-mon[84805]: osdmap e9: 1 total, 1 up, 1 in 2023-06-22T17:21:53.875 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:53 smithi101 ceph-mon[85507]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:21:53.875 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:53 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:21:53.875 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:53 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:21:53.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:53 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:53.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:53 smithi101 ceph-mon[85507]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-22T17:21:54.169 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:53 smithi133 ceph-mon[86268]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:21:54.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:53 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:21:54.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:53 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:21:54.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:53 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:54.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:53 smithi133 ceph-mon[86268]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-22T17:21:54.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:53 smithi039 ceph-mon[84805]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:21:54.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:53 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:21:54.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:53 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:21:54.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:53 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:21:54.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:53 smithi039 ceph-mon[84805]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-22T17:21:55.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:55 smithi101 ceph-mon[85507]: from='client.? 172.21.15.101:0/2500878286' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b8dbe3e-fd72-4f92-98a2-e212df5366c3"}]: dispatch 2023-06-22T17:21:55.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:55 smithi101 ceph-mon[85507]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b8dbe3e-fd72-4f92-98a2-e212df5366c3"}]: dispatch 2023-06-22T17:21:55.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:55 smithi101 ceph-mon[85507]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b8dbe3e-fd72-4f92-98a2-e212df5366c3"}]': finished 2023-06-22T17:21:55.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:55 smithi101 ceph-mon[85507]: osdmap e10: 2 total, 1 up, 2 in 2023-06-22T17:21:55.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:55 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-06-22T17:21:55.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:55 smithi133 ceph-mon[86268]: from='client.? 172.21.15.101:0/2500878286' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b8dbe3e-fd72-4f92-98a2-e212df5366c3"}]: dispatch 2023-06-22T17:21:55.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:55 smithi133 ceph-mon[86268]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b8dbe3e-fd72-4f92-98a2-e212df5366c3"}]: dispatch 2023-06-22T17:21:55.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:55 smithi133 ceph-mon[86268]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b8dbe3e-fd72-4f92-98a2-e212df5366c3"}]': finished 2023-06-22T17:21:55.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:55 smithi133 ceph-mon[86268]: osdmap e10: 2 total, 1 up, 2 in 2023-06-22T17:21:55.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:55 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-06-22T17:21:55.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:55 smithi039 ceph-mon[84805]: from='client.? 172.21.15.101:0/2500878286' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b8dbe3e-fd72-4f92-98a2-e212df5366c3"}]: dispatch 2023-06-22T17:21:55.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:55 smithi039 ceph-mon[84805]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b8dbe3e-fd72-4f92-98a2-e212df5366c3"}]: dispatch 2023-06-22T17:21:55.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:55 smithi039 ceph-mon[84805]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b8dbe3e-fd72-4f92-98a2-e212df5366c3"}]': finished 2023-06-22T17:21:55.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:55 smithi039 ceph-mon[84805]: osdmap e10: 2 total, 1 up, 2 in 2023-06-22T17:21:55.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:55 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-06-22T17:21:56.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:56 smithi101 ceph-mon[85507]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-22T17:21:56.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:56 smithi101 ceph-mon[85507]: from='client.? 172.21.15.101:0/1574386359' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:21:56.669 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:56 smithi133 ceph-mon[86268]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-22T17:21:56.669 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:56 smithi133 ceph-mon[86268]: from='client.? 172.21.15.101:0/1574386359' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:21:56.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:56 smithi039 ceph-mon[84805]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-22T17:21:56.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:56 smithi039 ceph-mon[84805]: from='client.? 172.21.15.101:0/1574386359' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:21:58.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:21:58 smithi101 ceph-mon[85507]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-22T17:21:58.669 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:21:58 smithi133 ceph-mon[86268]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-22T17:21:58.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:21:58 smithi039 ceph-mon[84805]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-22T17:22:00.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:00 smithi101 ceph-mon[85507]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-22T17:22:00.669 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:00 smithi133 ceph-mon[86268]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-22T17:22:00.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:00 smithi039 ceph-mon[84805]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-22T17:22:01.875 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:01 smithi101 ceph-mon[85507]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-22T17:22:01.919 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:01 smithi133 ceph-mon[86268]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-22T17:22:01.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:01 smithi039 ceph-mon[84805]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-22T17:22:02.875 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:02 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-06-22T17:22:02.875 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:02 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:02.875 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:02 smithi101 ceph-mon[85507]: Deploying daemon osd.1 on smithi101 2023-06-22T17:22:02.919 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-06-22T17:22:02.919 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:02.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:02 smithi133 ceph-mon[86268]: Deploying daemon osd.1 on smithi101 2023-06-22T17:22:02.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:02 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-06-22T17:22:02.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:02 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:02.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:02 smithi039 ceph-mon[84805]: Deploying daemon osd.1 on smithi101 2023-06-22T17:22:03.794 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:03 smithi101 ceph-mon[85507]: pgmap v48: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-22T17:22:03.919 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:03 smithi133 ceph-mon[86268]: pgmap v48: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-22T17:22:03.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:03 smithi039 ceph-mon[84805]: pgmap v48: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-22T17:22:06.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:06 smithi101 ceph-mon[85507]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-22T17:22:06.669 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:06 smithi133 ceph-mon[86268]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-22T17:22:06.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:06 smithi039 ceph-mon[84805]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-22T17:22:07.584 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:07 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:07.584 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:07 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:07.585 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:07 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:22:07.669 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:07 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:07.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:07 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:07.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:07 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:22:07.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:07 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:07.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:07 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:07.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:07 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:22:08.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:08 smithi101 ceph-mon[85507]: pgmap v50: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-22T17:22:08.669 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:08 smithi133 ceph-mon[86268]: pgmap v50: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-22T17:22:08.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:08 smithi039 ceph-mon[84805]: pgmap v50: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-22T17:22:09.307 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:09.307+0000 7f9445ffb700 1 -- 172.21.15.101:0/1265751273 <== mgr.14150 v2:172.21.15.39:6800/886078282 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f945002b130 con 0x7f943c09b670 2023-06-22T17:22:09.307 INFO:teuthology.orchestra.run.smithi101.stdout:Created osd(s) 1 on host 'smithi101' 2023-06-22T17:22:09.309 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:09.309+0000 7f9457a39700 1 -- 172.21.15.101:0/1265751273 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f943c09b670 msgr2=0x7f943c09db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:22:09.310 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:09.309+0000 7f9457a39700 1 --2- 172.21.15.101:0/1265751273 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f943c09b670 0x7f943c09db20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f944c006fd0 tx=0x7f944c008040 comp rx=0 tx=0).stop 2023-06-22T17:22:09.310 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:09.309+0000 7f9457a39700 1 -- 172.21.15.101:0/1265751273 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f94500fde90 msgr2=0x7f94501144c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:22:09.310 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:09.309+0000 7f9457a39700 1 --2- 172.21.15.101:0/1265751273 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f94500fde90 0x7f94501144c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f944000b810 tx=0x7f9440009250 comp rx=0 tx=0).stop 2023-06-22T17:22:09.310 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:09.309+0000 7f9457a39700 1 -- 172.21.15.101:0/1265751273 shutdown_connections 2023-06-22T17:22:09.310 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:09.309+0000 7f9457a39700 1 --2- 172.21.15.101:0/1265751273 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f943c09b670 0x7f943c09db20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:22:09.310 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:09.309+0000 7f9457a39700 1 --2- 172.21.15.101:0/1265751273 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94500fe820 0x7f945011b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:22:09.310 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:09.309+0000 7f9457a39700 1 --2- 172.21.15.101:0/1265751273 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f94500fcc80 0x7f9450113f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:22:09.311 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:09.309+0000 7f9457a39700 1 --2- 172.21.15.101:0/1265751273 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f94500fde90 0x7f94501144c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:22:09.311 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:09.309+0000 7f9457a39700 1 -- 172.21.15.101:0/1265751273 >> 172.21.15.101:0/1265751273 conn(0x7f94500f83a0 msgr2=0x7f94500ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:22:09.311 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:09.309+0000 7f9457a39700 1 -- 172.21.15.101:0/1265751273 shutdown_connections 2023-06-22T17:22:09.311 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:09.309+0000 7f9457a39700 1 -- 172.21.15.101:0/1265751273 wait complete. 2023-06-22T17:22:09.755 DEBUG:teuthology.orchestra.run.smithi101:osd.1> sudo journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.1.service 2023-06-22T17:22:09.757 INFO:tasks.cephadm:Deploying osd.2 on smithi133 with /dev/vg_nvme/lv_4... 2023-06-22T17:22:09.757 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-06-22T17:22:09.778 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:09 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:09.778 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:09 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:09.779 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:09 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:09.779 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:09 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:22:09.779 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:09 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:09.779 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:09 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:09.780 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:09 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:09.780 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:09 smithi101 ceph-mon[85507]: pgmap v51: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-22T17:22:09.780 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:09 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:22:09.784 INFO:journalctl@ceph.osd.1.smithi101.stdout:-- Logs begin at Thu 2023-06-22 17:06:34 UTC. -- 2023-06-22T17:22:09.919 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:09 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:09.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:09 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:09.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:09 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:09.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:09 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:22:09.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:09 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:09.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:09 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:09.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:09 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:09.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:09 smithi133 ceph-mon[86268]: pgmap v51: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-22T17:22:09.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:09 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:22:09.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:09 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:09.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:09 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:09.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:09 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:09.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:09 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:22:09.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:09 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:09.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:09 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:09.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:09 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:09.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:09 smithi039 ceph-mon[84805]: pgmap v51: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-22T17:22:09.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:09 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:22:10.347 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.c/config 2023-06-22T17:22:10.625 INFO:journalctl@ceph.osd.1.smithi101.stdout:Jun 22 17:22:10 smithi101 conmon[91956]: 2023-06-22T17:22:10.284+0000 7f8024ace540 -1 osd.1 0 log_to_monitors true 2023-06-22T17:22:11.375 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:11 smithi101 ceph-mon[85507]: from='osd.1 [v2:172.21.15.101:6800/2708845767,v1:172.21.15.101:6801/2708845767]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-06-22T17:22:11.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:11 smithi101 ceph-mon[85507]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-06-22T17:22:11.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:11 smithi133 ceph-mon[86268]: from='osd.1 [v2:172.21.15.101:6800/2708845767,v1:172.21.15.101:6801/2708845767]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-06-22T17:22:11.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:11 smithi133 ceph-mon[86268]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-06-22T17:22:11.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:11 smithi039 ceph-mon[84805]: from='osd.1 [v2:172.21.15.101:6800/2708845767,v1:172.21.15.101:6801/2708845767]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-06-22T17:22:11.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:11 smithi039 ceph-mon[84805]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-06-22T17:22:11.890 INFO:teuthology.orchestra.run.smithi133.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-06-22T17:22:11.890 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-06-22T17:22:11.890 INFO:teuthology.orchestra.run.smithi133.stderr: stderr: 10+0 records in 2023-06-22T17:22:11.890 INFO:teuthology.orchestra.run.smithi133.stderr:10+0 records out 2023-06-22T17:22:11.891 INFO:teuthology.orchestra.run.smithi133.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0147719 s, 710 MB/s 2023-06-22T17:22:11.891 INFO:teuthology.orchestra.run.smithi133.stderr:--> Zapping successful for: 2023-06-22T17:22:12.292 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph orch daemon add osd smithi133:vg_nvme/lv_4 2023-06-22T17:22:12.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:12 smithi101 ceph-mon[85507]: pgmap v52: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-22T17:22:12.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:12 smithi101 ceph-mon[85507]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-06-22T17:22:12.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:12 smithi101 ceph-mon[85507]: osdmap e11: 2 total, 1 up, 2 in 2023-06-22T17:22:12.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:12 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-06-22T17:22:12.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:12 smithi101 ceph-mon[85507]: from='osd.1 [v2:172.21.15.101:6800/2708845767,v1:172.21.15.101:6801/2708845767]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-06-22T17:22:12.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:12 smithi101 ceph-mon[85507]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-06-22T17:22:12.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:12 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:12.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:12 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:12.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:12 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:22:12.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:12 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:12.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:12 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:12.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:12 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:22:12.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:12 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:12.628 INFO:journalctl@ceph.osd.1.smithi101.stdout:Jun 22 17:22:12 smithi101 conmon[91956]: 2023-06-22T17:22:12.313+0000 7f801b2b3700 -1 osd.1 0 waiting for initial osdmap 2023-06-22T17:22:12.628 INFO:journalctl@ceph.osd.1.smithi101.stdout:Jun 22 17:22:12 smithi101 conmon[91956]: 2023-06-22T17:22:12.317+0000 7f80160ae700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-22T17:22:12.669 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:12 smithi133 ceph-mon[86268]: pgmap v52: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-22T17:22:12.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:12 smithi133 ceph-mon[86268]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-06-22T17:22:12.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:12 smithi133 ceph-mon[86268]: osdmap e11: 2 total, 1 up, 2 in 2023-06-22T17:22:12.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:12 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-06-22T17:22:12.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:12 smithi133 ceph-mon[86268]: from='osd.1 [v2:172.21.15.101:6800/2708845767,v1:172.21.15.101:6801/2708845767]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-06-22T17:22:12.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:12 smithi133 ceph-mon[86268]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-06-22T17:22:12.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:12 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:12.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:12 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:12.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:12 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:22:12.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:12 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:12.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:12 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:12.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:12 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:22:12.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:12 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:12.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:12 smithi039 ceph-mon[84805]: pgmap v52: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-22T17:22:12.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:12 smithi039 ceph-mon[84805]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-06-22T17:22:12.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:12 smithi039 ceph-mon[84805]: osdmap e11: 2 total, 1 up, 2 in 2023-06-22T17:22:12.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:12 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-06-22T17:22:12.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:12 smithi039 ceph-mon[84805]: from='osd.1 [v2:172.21.15.101:6800/2708845767,v1:172.21.15.101:6801/2708845767]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-06-22T17:22:12.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:12 smithi039 ceph-mon[84805]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-06-22T17:22:12.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:12 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:12.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:12 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:12.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:12 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:22:12.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:12 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:12.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:12 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:12.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:12 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:22:12.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:12 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:12.872 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.c/config 2023-06-22T17:22:13.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:13 smithi133 ceph-mon[86268]: purged_snaps scrub starts 2023-06-22T17:22:13.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:13 smithi133 ceph-mon[86268]: purged_snaps scrub ok 2023-06-22T17:22:13.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:13 smithi133 ceph-mon[86268]: Detected new or changed devices on smithi101 2023-06-22T17:22:13.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:13 smithi133 ceph-mon[86268]: Adjusting osd_memory_target on smithi101 to 17168M 2023-06-22T17:22:13.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:13 smithi133 ceph-mon[86268]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2023-06-22T17:22:13.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:13 smithi133 ceph-mon[86268]: osdmap e12: 2 total, 1 up, 2 in 2023-06-22T17:22:13.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:13 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-06-22T17:22:13.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:13 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-06-22T17:22:13.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:13 smithi101 ceph-mon[85507]: purged_snaps scrub starts 2023-06-22T17:22:13.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:13 smithi101 ceph-mon[85507]: purged_snaps scrub ok 2023-06-22T17:22:13.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:13 smithi101 ceph-mon[85507]: Detected new or changed devices on smithi101 2023-06-22T17:22:13.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:13 smithi101 ceph-mon[85507]: Adjusting osd_memory_target on smithi101 to 17168M 2023-06-22T17:22:13.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:13 smithi101 ceph-mon[85507]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2023-06-22T17:22:13.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:13 smithi101 ceph-mon[85507]: osdmap e12: 2 total, 1 up, 2 in 2023-06-22T17:22:13.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:13 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-06-22T17:22:13.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:13 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-06-22T17:22:13.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:13 smithi039 ceph-mon[84805]: purged_snaps scrub starts 2023-06-22T17:22:13.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:13 smithi039 ceph-mon[84805]: purged_snaps scrub ok 2023-06-22T17:22:13.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:13 smithi039 ceph-mon[84805]: Detected new or changed devices on smithi101 2023-06-22T17:22:13.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:13 smithi039 ceph-mon[84805]: Adjusting osd_memory_target on smithi101 to 17168M 2023-06-22T17:22:13.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:13 smithi039 ceph-mon[84805]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2023-06-22T17:22:13.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:13 smithi039 ceph-mon[84805]: osdmap e12: 2 total, 1 up, 2 in 2023-06-22T17:22:13.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:13 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-06-22T17:22:13.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:13 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-06-22T17:22:14.170 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.169+0000 7f5be2823700 1 -- 172.21.15.133:0/2509037216 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5bdc0fcda0 msgr2=0x7f5bdc0fd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:22:14.171 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.169+0000 7f5be2823700 1 --2- 172.21.15.133:0/2509037216 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5bdc0fcda0 0x7f5bdc0fd210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5bd00049c0 tx=0x7f5bd0009e30 comp rx=0 tx=0).stop 2023-06-22T17:22:14.171 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.169+0000 7f5be2823700 1 -- 172.21.15.133:0/2509037216 shutdown_connections 2023-06-22T17:22:14.171 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.169+0000 7f5be2823700 1 --2- 172.21.15.133:0/2509037216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5bdc0fd750 0x7f5bdc103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:22:14.171 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.169+0000 7f5be2823700 1 --2- 172.21.15.133:0/2509037216 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5bdc0fbaf0 0x7f5bdc0fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:22:14.171 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.169+0000 7f5be2823700 1 --2- 172.21.15.133:0/2509037216 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5bdc0fcda0 0x7f5bdc0fd210 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:22:14.171 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.169+0000 7f5be2823700 1 -- 172.21.15.133:0/2509037216 >> 172.21.15.133:0/2509037216 conn(0x7f5bdc0f7210 msgr2=0x7f5bdc0f9680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:22:14.172 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.169+0000 7f5be2823700 1 -- 172.21.15.133:0/2509037216 shutdown_connections 2023-06-22T17:22:14.172 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.169+0000 7f5be2823700 1 -- 172.21.15.133:0/2509037216 wait complete. 2023-06-22T17:22:14.172 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.169+0000 7f5be2823700 1 Processor -- start 2023-06-22T17:22:14.174 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.170+0000 7f5be2823700 1 -- start start 2023-06-22T17:22:14.174 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.170+0000 7f5be2823700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5bdc0fbaf0 0x7f5bdc065f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:22:14.174 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.170+0000 7f5be2823700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5bdc0fcda0 0x7f5bdc1127f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:22:14.175 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.170+0000 7f5be2823700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5bdc0fd750 0x7f5bdc114e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:22:14.175 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.170+0000 7f5be2823700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc1155d0 con 0x7f5bdc0fbaf0 2023-06-22T17:22:14.175 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.170+0000 7f5be2823700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc115740 con 0x7f5bdc0fcda0 2023-06-22T17:22:14.175 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.170+0000 7f5be2823700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc115900 con 0x7f5bdc0fd750 2023-06-22T17:22:14.175 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.170+0000 7f5be0dc0700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5bdc0fd750 0x7f5bdc114e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:22:14.175 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.170+0000 7f5be0dc0700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5bdc0fd750 0x7f5bdc114e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:57500/0 (socket says 172.21.15.133:57500) 2023-06-22T17:22:14.175 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.170+0000 7f5be0dc0700 1 -- 172.21.15.133:0/1419334973 learned_addr learned my addr 172.21.15.133:0/1419334973 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-06-22T17:22:14.175 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.170+0000 7f5bdbfff700 1 --2- 172.21.15.133:0/1419334973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5bdc0fbaf0 0x7f5bdc065f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:22:14.176 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.170+0000 7f5bdb7fe700 1 --2- 172.21.15.133:0/1419334973 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5bdc0fcda0 0x7f5bdc1127f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:22:14.176 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.170+0000 7f5be0dc0700 1 -- 172.21.15.133:0/1419334973 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5bdc0fcda0 msgr2=0x7f5bdc1127f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:22:14.176 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.170+0000 7f5be0dc0700 1 --2- 172.21.15.133:0/1419334973 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5bdc0fcda0 0x7f5bdc1127f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:22:14.176 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.170+0000 7f5be0dc0700 1 -- 172.21.15.133:0/1419334973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5bdc0fbaf0 msgr2=0x7f5bdc065f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:22:14.176 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.170+0000 7f5be0dc0700 1 --2- 172.21.15.133:0/1419334973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5bdc0fbaf0 0x7f5bdc065f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:22:14.176 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.170+0000 7f5be0dc0700 1 -- 172.21.15.133:0/1419334973 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd0005040 con 0x7f5bdc0fd750 2023-06-22T17:22:14.176 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.170+0000 7f5be0dc0700 1 --2- 172.21.15.133:0/1419334973 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5bdc0fd750 0x7f5bdc114e30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4001d80 tx=0x7f5bd4009ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:22:14.177 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.171+0000 7f5bd8ff9700 1 -- 172.21.15.133:0/1419334973 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd40048f0 con 0x7f5bdc0fd750 2023-06-22T17:22:14.177 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.171+0000 7f5bd8ff9700 1 -- 172.21.15.133:0/1419334973 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f5bd400ed90 con 0x7f5bdc0fd750 2023-06-22T17:22:14.177 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.171+0000 7f5be2823700 1 -- 172.21.15.133:0/1419334973 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bdc115be0 con 0x7f5bdc0fd750 2023-06-22T17:22:14.177 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.171+0000 7f5bd8ff9700 1 -- 172.21.15.133:0/1419334973 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd4009720 con 0x7f5bdc0fd750 2023-06-22T17:22:14.177 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.171+0000 7f5be2823700 1 -- 172.21.15.133:0/1419334973 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bdc116080 con 0x7f5bdc0fd750 2023-06-22T17:22:14.177 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.172+0000 7f5be2823700 1 -- 172.21.15.133:0/1419334973 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bdc102a50 con 0x7f5bdc0fd750 2023-06-22T17:22:14.177 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.173+0000 7f5bd8ff9700 1 -- 172.21.15.133:0/1419334973 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 15) v1 ==== 129145+0+0 (secure 0 0 0) 0x7f5bd400e8b0 con 0x7f5bdc0fd750 2023-06-22T17:22:14.177 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.173+0000 7f5bd8ff9700 1 --2- 172.21.15.133:0/1419334973 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f5bc409b5f0 0x7f5bc409daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:22:14.178 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.173+0000 7f5bd8ff9700 1 -- 172.21.15.133:0/1419334973 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2305+0+0 (secure 0 0 0) 0x7f5bd40c35c0 con 0x7f5bdc0fd750 2023-06-22T17:22:14.178 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.173+0000 7f5bdbfff700 1 --2- 172.21.15.133:0/1419334973 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f5bc409b5f0 0x7f5bc409daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:22:14.178 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.174+0000 7f5bdbfff700 1 --2- 172.21.15.133:0/1419334973 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f5bc409b5f0 0x7f5bc409daa0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc009bf0 tx=0x7f5bcc0091a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:22:14.179 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.177+0000 7f5bd8ff9700 1 -- 172.21.15.133:0/1419334973 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f5bd4090670 con 0x7f5bdc0fd750 2023-06-22T17:22:14.370 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:14.368+0000 7f5be2823700 1 -- 172.21.15.133:0/1419334973 --> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f5bdc02b2c0 con 0x7f5bc409b5f0 2023-06-22T17:22:14.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:14 smithi101 ceph-mon[85507]: pgmap v55: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-22T17:22:14.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:14 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-06-22T17:22:14.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:14 smithi101 ceph-mon[85507]: osd.1 [v2:172.21.15.101:6800/2708845767,v1:172.21.15.101:6801/2708845767] boot 2023-06-22T17:22:14.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:14 smithi101 ceph-mon[85507]: osdmap e13: 2 total, 2 up, 2 in 2023-06-22T17:22:14.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:14 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-06-22T17:22:14.669 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:14 smithi133 ceph-mon[86268]: pgmap v55: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-22T17:22:14.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:14 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-06-22T17:22:14.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:14 smithi133 ceph-mon[86268]: osd.1 [v2:172.21.15.101:6800/2708845767,v1:172.21.15.101:6801/2708845767] boot 2023-06-22T17:22:14.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:14 smithi133 ceph-mon[86268]: osdmap e13: 2 total, 2 up, 2 in 2023-06-22T17:22:14.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:14 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-06-22T17:22:14.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:14 smithi039 ceph-mon[84805]: pgmap v55: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-22T17:22:14.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:14 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-06-22T17:22:14.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:14 smithi039 ceph-mon[84805]: osd.1 [v2:172.21.15.101:6800/2708845767,v1:172.21.15.101:6801/2708845767] boot 2023-06-22T17:22:14.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:14 smithi039 ceph-mon[84805]: osdmap e13: 2 total, 2 up, 2 in 2023-06-22T17:22:14.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:14 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-06-22T17:22:15.579 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:15 smithi133 ceph-mon[86268]: osdmap e14: 2 total, 2 up, 2 in 2023-06-22T17:22:15.579 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:15 smithi133 ceph-mon[86268]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:22:15.579 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:15 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:22:15.579 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:15 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:22:15.579 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:15 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:15.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:15 smithi101 ceph-mon[85507]: osdmap e14: 2 total, 2 up, 2 in 2023-06-22T17:22:15.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:15 smithi101 ceph-mon[85507]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:22:15.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:15 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:22:15.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:15 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:22:15.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:15 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:15.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:15 smithi039 ceph-mon[84805]: osdmap e14: 2 total, 2 up, 2 in 2023-06-22T17:22:15.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:15 smithi039 ceph-mon[84805]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:22:15.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:15 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:22:15.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:15 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:22:15.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:15 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:16.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:16 smithi101 ceph-mon[85507]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-06-22T17:22:16.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:16 smithi133 ceph-mon[86268]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-06-22T17:22:16.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:16 smithi039 ceph-mon[84805]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-06-22T17:22:17.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:17 smithi101 ceph-mon[85507]: from='client.? 172.21.15.133:0/2342141646' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b25ade6-b5e2-4194-92c7-3baf8c4e9560"}]: dispatch 2023-06-22T17:22:17.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:17 smithi101 ceph-mon[85507]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b25ade6-b5e2-4194-92c7-3baf8c4e9560"}]: dispatch 2023-06-22T17:22:17.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:17 smithi101 ceph-mon[85507]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b25ade6-b5e2-4194-92c7-3baf8c4e9560"}]': finished 2023-06-22T17:22:17.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:17 smithi101 ceph-mon[85507]: osdmap e15: 3 total, 2 up, 3 in 2023-06-22T17:22:17.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:17 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-06-22T17:22:17.669 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:17 smithi133 ceph-mon[86268]: from='client.? 172.21.15.133:0/2342141646' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b25ade6-b5e2-4194-92c7-3baf8c4e9560"}]: dispatch 2023-06-22T17:22:17.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:17 smithi133 ceph-mon[86268]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b25ade6-b5e2-4194-92c7-3baf8c4e9560"}]: dispatch 2023-06-22T17:22:17.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:17 smithi133 ceph-mon[86268]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b25ade6-b5e2-4194-92c7-3baf8c4e9560"}]': finished 2023-06-22T17:22:17.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:17 smithi133 ceph-mon[86268]: osdmap e15: 3 total, 2 up, 3 in 2023-06-22T17:22:17.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:17 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-06-22T17:22:17.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:17 smithi039 ceph-mon[84805]: from='client.? 172.21.15.133:0/2342141646' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b25ade6-b5e2-4194-92c7-3baf8c4e9560"}]: dispatch 2023-06-22T17:22:17.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:17 smithi039 ceph-mon[84805]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b25ade6-b5e2-4194-92c7-3baf8c4e9560"}]: dispatch 2023-06-22T17:22:17.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:17 smithi039 ceph-mon[84805]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b25ade6-b5e2-4194-92c7-3baf8c4e9560"}]': finished 2023-06-22T17:22:17.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:17 smithi039 ceph-mon[84805]: osdmap e15: 3 total, 2 up, 3 in 2023-06-22T17:22:17.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:17 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-06-22T17:22:18.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:18 smithi101 ceph-mon[85507]: pgmap v60: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-06-22T17:22:18.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:18 smithi101 ceph-mon[85507]: from='client.? 172.21.15.133:0/3989612860' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:22:18.669 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:18 smithi133 ceph-mon[86268]: pgmap v60: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-06-22T17:22:18.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:18 smithi133 ceph-mon[86268]: from='client.? 172.21.15.133:0/3989612860' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:22:18.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:18 smithi039 ceph-mon[84805]: pgmap v60: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-06-22T17:22:18.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:18 smithi039 ceph-mon[84805]: from='client.? 172.21.15.133:0/3989612860' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:22:20.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:20 smithi101 ceph-mon[85507]: pgmap v61: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-06-22T17:22:20.653 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:20 smithi133 ceph-mon[86268]: pgmap v61: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-06-22T17:22:20.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:20 smithi039 ceph-mon[84805]: pgmap v61: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-06-22T17:22:21.820 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:21 smithi133 ceph-mon[86268]: pgmap v62: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-06-22T17:22:21.875 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:21 smithi101 ceph-mon[85507]: pgmap v62: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-06-22T17:22:21.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:21 smithi039 ceph-mon[84805]: pgmap v62: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-06-22T17:22:24.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:24 smithi133 ceph-mon[86268]: pgmap v63: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-22T17:22:24.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:24 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-06-22T17:22:24.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:24 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:24.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:24 smithi133 ceph-mon[86268]: Deploying daemon osd.2 on smithi133 2023-06-22T17:22:24.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:24 smithi101 ceph-mon[85507]: pgmap v63: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-22T17:22:24.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:24 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-06-22T17:22:24.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:24 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:24.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:24 smithi101 ceph-mon[85507]: Deploying daemon osd.2 on smithi133 2023-06-22T17:22:24.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:24 smithi039 ceph-mon[84805]: pgmap v63: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-22T17:22:24.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:24 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-06-22T17:22:24.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:24 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:24.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:24 smithi039 ceph-mon[84805]: Deploying daemon osd.2 on smithi133 2023-06-22T17:22:26.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:26 smithi101 ceph-mon[85507]: pgmap v64: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-22T17:22:26.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:26 smithi133 ceph-mon[86268]: pgmap v64: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-22T17:22:26.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:26 smithi039 ceph-mon[84805]: pgmap v64: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-22T17:22:28.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:28 smithi101 ceph-mon[85507]: pgmap v65: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-22T17:22:28.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:28 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:28.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:28 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:28.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:28 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:22:28.669 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:28 smithi133 ceph-mon[86268]: pgmap v65: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-22T17:22:28.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:28 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:28.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:28 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:28.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:28 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:22:28.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:28 smithi039 ceph-mon[84805]: pgmap v65: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-22T17:22:28.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:28 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:28.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:28 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:28.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:28 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:22:30.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:30 smithi133 ceph-mon[86268]: pgmap v66: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-22T17:22:30.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:30 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:30.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:30 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:30.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:30 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:30.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:30 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:22:30.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:30 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:30.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:30 smithi039 ceph-mon[84805]: pgmap v66: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-22T17:22:30.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:30 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:30.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:30 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:30.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:30 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:30.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:30 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:22:30.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:30 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:30.829 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:30.828+0000 7f5bd8ff9700 1 -- 172.21.15.133:0/1419334973 <== mgr.14150 v2:172.21.15.39:6800/886078282 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5bdc02b2c0 con 0x7f5bc409b5f0 2023-06-22T17:22:30.830 INFO:teuthology.orchestra.run.smithi133.stdout:Created osd(s) 2 on host 'smithi133' 2023-06-22T17:22:30.832 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:30.831+0000 7f5be2823700 1 -- 172.21.15.133:0/1419334973 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f5bc409b5f0 msgr2=0x7f5bc409daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:22:30.832 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:30.831+0000 7f5be2823700 1 --2- 172.21.15.133:0/1419334973 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f5bc409b5f0 0x7f5bc409daa0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc009bf0 tx=0x7f5bcc0091a0 comp rx=0 tx=0).stop 2023-06-22T17:22:30.832 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:30.831+0000 7f5be2823700 1 -- 172.21.15.133:0/1419334973 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5bdc0fd750 msgr2=0x7f5bdc114e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:22:30.833 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:30.831+0000 7f5be2823700 1 --2- 172.21.15.133:0/1419334973 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5bdc0fd750 0x7f5bdc114e30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4001d80 tx=0x7f5bd4009ee0 comp rx=0 tx=0).stop 2023-06-22T17:22:30.833 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:30.831+0000 7f5be2823700 1 -- 172.21.15.133:0/1419334973 shutdown_connections 2023-06-22T17:22:30.833 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:30.831+0000 7f5be2823700 1 --2- 172.21.15.133:0/1419334973 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f5bc409b5f0 0x7f5bc409daa0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:22:30.833 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:30.831+0000 7f5be2823700 1 --2- 172.21.15.133:0/1419334973 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5bdc0fd750 0x7f5bdc114e30 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:22:30.833 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:30.831+0000 7f5be2823700 1 --2- 172.21.15.133:0/1419334973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5bdc0fbaf0 0x7f5bdc065f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:22:30.833 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:30.831+0000 7f5be2823700 1 --2- 172.21.15.133:0/1419334973 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5bdc0fcda0 0x7f5bdc1127f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:22:30.833 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:30.831+0000 7f5be2823700 1 -- 172.21.15.133:0/1419334973 >> 172.21.15.133:0/1419334973 conn(0x7f5bdc0f7210 msgr2=0x7f5bdc0f9480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:22:30.834 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:30.831+0000 7f5be2823700 1 -- 172.21.15.133:0/1419334973 shutdown_connections 2023-06-22T17:22:30.834 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:22:30.831+0000 7f5be2823700 1 -- 172.21.15.133:0/1419334973 wait complete. 2023-06-22T17:22:30.875 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:30 smithi101 ceph-mon[85507]: pgmap v66: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-22T17:22:30.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:30 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:30.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:30 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:30.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:30 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:30.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:30 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:22:30.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:30 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:31.232 DEBUG:teuthology.orchestra.run.smithi133:osd.2> sudo journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.2.service 2023-06-22T17:22:31.235 INFO:tasks.cephadm:Deploying osd.3 on smithi039 with /dev/vg_nvme/lv_3... 2023-06-22T17:22:31.235 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-06-22T17:22:31.261 INFO:journalctl@ceph.osd.2.smithi133.stdout:-- Logs begin at Thu 2023-06-22 17:06:34 UTC. -- 2023-06-22T17:22:31.845 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-22T17:22:32.064 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:31 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:32.065 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:31 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:32.065 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:31 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:22:32.065 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:31 smithi039 ceph-mon[84805]: pgmap v67: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-22T17:22:32.068 INFO:journalctl@ceph.osd.2.smithi133.stdout:Jun 22 17:22:31 smithi133 conmon[93974]: 2023-06-22T17:22:31.878+0000 7f3194ef4540 -1 osd.2 0 log_to_monitors true 2023-06-22T17:22:32.068 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:31 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:32.068 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:31 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:32.069 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:31 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:22:32.069 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:31 smithi133 ceph-mon[86268]: pgmap v67: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-22T17:22:32.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:31 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:32.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:31 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:32.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:31 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:22:32.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:31 smithi101 ceph-mon[85507]: pgmap v67: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-22T17:22:33.125 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:32 smithi101 ceph-mon[85507]: from='osd.2 [v2:172.21.15.133:6800/3632902853,v1:172.21.15.133:6801/3632902853]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-06-22T17:22:33.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:32 smithi101 ceph-mon[85507]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-06-22T17:22:33.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:32 smithi133 ceph-mon[86268]: from='osd.2 [v2:172.21.15.133:6800/3632902853,v1:172.21.15.133:6801/3632902853]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-06-22T17:22:33.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:32 smithi133 ceph-mon[86268]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-06-22T17:22:33.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:32 smithi039 ceph-mon[84805]: from='osd.2 [v2:172.21.15.133:6800/3632902853,v1:172.21.15.133:6801/3632902853]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-06-22T17:22:33.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:32 smithi039 ceph-mon[84805]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-06-22T17:22:33.361 INFO:teuthology.orchestra.run.smithi039.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-06-22T17:22:33.362 INFO:teuthology.orchestra.run.smithi039.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-06-22T17:22:33.362 INFO:teuthology.orchestra.run.smithi039.stderr: stderr: 10+0 records in 2023-06-22T17:22:33.362 INFO:teuthology.orchestra.run.smithi039.stderr:10+0 records out 2023-06-22T17:22:33.362 INFO:teuthology.orchestra.run.smithi039.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.010364 s, 1.0 GB/s 2023-06-22T17:22:33.362 INFO:teuthology.orchestra.run.smithi039.stderr:--> Zapping successful for: 2023-06-22T17:22:33.794 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph orch daemon add osd smithi039:vg_nvme/lv_3 2023-06-22T17:22:34.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:33 smithi101 ceph-mon[85507]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-06-22T17:22:34.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:33 smithi101 ceph-mon[85507]: osdmap e16: 3 total, 2 up, 3 in 2023-06-22T17:22:34.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:33 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-06-22T17:22:34.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:33 smithi101 ceph-mon[85507]: from='osd.2 [v2:172.21.15.133:6800/3632902853,v1:172.21.15.133:6801/3632902853]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2023-06-22T17:22:34.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:33 smithi101 ceph-mon[85507]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2023-06-22T17:22:34.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:33 smithi101 ceph-mon[85507]: pgmap v69: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-22T17:22:34.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:33 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:34.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:33 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:34.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:33 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:22:34.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:33 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:34.128 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:33 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:34.128 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:33 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:22:34.128 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:33 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:34.170 INFO:journalctl@ceph.osd.2.smithi133.stdout:Jun 22 17:22:33 smithi133 conmon[93974]: 2023-06-22T17:22:33.852+0000 7f318b6d9700 -1 osd.2 0 waiting for initial osdmap 2023-06-22T17:22:34.170 INFO:journalctl@ceph.osd.2.smithi133.stdout:Jun 22 17:22:33 smithi133 conmon[93974]: 2023-06-22T17:22:33.856+0000 7f3187cd7700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-22T17:22:34.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:33 smithi133 ceph-mon[86268]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-06-22T17:22:34.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:33 smithi133 ceph-mon[86268]: osdmap e16: 3 total, 2 up, 3 in 2023-06-22T17:22:34.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:33 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-06-22T17:22:34.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:33 smithi133 ceph-mon[86268]: from='osd.2 [v2:172.21.15.133:6800/3632902853,v1:172.21.15.133:6801/3632902853]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2023-06-22T17:22:34.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:33 smithi133 ceph-mon[86268]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2023-06-22T17:22:34.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:33 smithi133 ceph-mon[86268]: pgmap v69: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-22T17:22:34.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:33 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:34.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:33 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:34.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:33 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:22:34.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:33 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:34.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:33 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:34.172 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:33 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:22:34.172 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:33 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:34.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:33 smithi039 ceph-mon[84805]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-06-22T17:22:34.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:33 smithi039 ceph-mon[84805]: osdmap e16: 3 total, 2 up, 3 in 2023-06-22T17:22:34.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:33 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-06-22T17:22:34.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:33 smithi039 ceph-mon[84805]: from='osd.2 [v2:172.21.15.133:6800/3632902853,v1:172.21.15.133:6801/3632902853]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2023-06-22T17:22:34.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:33 smithi039 ceph-mon[84805]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2023-06-22T17:22:34.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:33 smithi039 ceph-mon[84805]: pgmap v69: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-22T17:22:34.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:33 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:34.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:33 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:34.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:33 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:22:34.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:33 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:34.185 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:33 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:34.185 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:33 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:22:34.185 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:33 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:34.389 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-22T17:22:34.851 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:34 smithi039 ceph-mon[84805]: Detected new or changed devices on smithi133 2023-06-22T17:22:34.851 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:34 smithi039 ceph-mon[84805]: Adjusting osd_memory_target on smithi133 to 17168M 2023-06-22T17:22:34.852 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:34 smithi039 ceph-mon[84805]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]': finished 2023-06-22T17:22:34.852 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:34 smithi039 ceph-mon[84805]: osdmap e17: 3 total, 2 up, 3 in 2023-06-22T17:22:34.852 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:34 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-06-22T17:22:34.852 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:34 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-06-22T17:22:35.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:34 smithi101 ceph-mon[85507]: Detected new or changed devices on smithi133 2023-06-22T17:22:35.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:34 smithi101 ceph-mon[85507]: Adjusting osd_memory_target on smithi133 to 17168M 2023-06-22T17:22:35.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:34 smithi101 ceph-mon[85507]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]': finished 2023-06-22T17:22:35.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:34 smithi101 ceph-mon[85507]: osdmap e17: 3 total, 2 up, 3 in 2023-06-22T17:22:35.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:34 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-06-22T17:22:35.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:34 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-06-22T17:22:35.169 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:34 smithi133 ceph-mon[86268]: Detected new or changed devices on smithi133 2023-06-22T17:22:35.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:34 smithi133 ceph-mon[86268]: Adjusting osd_memory_target on smithi133 to 17168M 2023-06-22T17:22:35.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:34 smithi133 ceph-mon[86268]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]': finished 2023-06-22T17:22:35.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:34 smithi133 ceph-mon[86268]: osdmap e17: 3 total, 2 up, 3 in 2023-06-22T17:22:35.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:34 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-06-22T17:22:35.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:34 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-06-22T17:22:35.800 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.799+0000 7f5a35866700 1 -- 172.21.15.39:0/2052841530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a30102870 msgr2=0x7f5a30104d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:22:35.801 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.799+0000 7f5a35866700 1 --2- 172.21.15.39:0/2052841530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a30102870 0x7f5a30104d20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5a28009d60 tx=0x7f5a28004d10 comp rx=0 tx=0).stop 2023-06-22T17:22:35.801 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.799+0000 7f5a35866700 1 -- 172.21.15.39:0/2052841530 shutdown_connections 2023-06-22T17:22:35.801 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.799+0000 7f5a35866700 1 --2- 172.21.15.39:0/2052841530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a30102870 0x7f5a30104d20 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:22:35.801 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.799+0000 7f5a35866700 1 --2- 172.21.15.39:0/2052841530 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5a30066110 0x7f5a300ff7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:22:35.801 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.799+0000 7f5a35866700 1 --2- 172.21.15.39:0/2052841530 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5a300ffd20 0x7f5a30102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:22:35.801 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.799+0000 7f5a35866700 1 -- 172.21.15.39:0/2052841530 >> 172.21.15.39:0/2052841530 conn(0x7f5a300f7210 msgr2=0x7f5a300f9680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:22:35.802 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.799+0000 7f5a35866700 1 -- 172.21.15.39:0/2052841530 shutdown_connections 2023-06-22T17:22:35.802 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.799+0000 7f5a35866700 1 -- 172.21.15.39:0/2052841530 wait complete. 2023-06-22T17:22:35.803 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.800+0000 7f5a35866700 1 Processor -- start 2023-06-22T17:22:35.803 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.800+0000 7f5a35866700 1 -- start start 2023-06-22T17:22:35.803 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.800+0000 7f5a35866700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a30066110 0x7f5a300fd080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:22:35.804 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.800+0000 7f5a35866700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5a300ffd20 0x7f5a300fb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:22:35.804 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.801+0000 7f5a35866700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5a30102870 0x7f5a300fbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:22:35.804 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.801+0000 7f5a35866700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a300fc390 con 0x7f5a30066110 2023-06-22T17:22:35.804 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.801+0000 7f5a35866700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a300fc500 con 0x7f5a30102870 2023-06-22T17:22:35.804 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.801+0000 7f5a35866700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a301178b0 con 0x7f5a300ffd20 2023-06-22T17:22:35.805 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.801+0000 7f5a2effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a30066110 0x7f5a300fd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:22:35.805 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.801+0000 7f5a2effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a30066110 0x7f5a300fd080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51728/0 (socket says 172.21.15.39:51728) 2023-06-22T17:22:35.805 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.801+0000 7f5a2effd700 1 -- 172.21.15.39:0/2544272044 learned_addr learned my addr 172.21.15.39:0/2544272044 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:22:35.805 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.801+0000 7f5a2e7fc700 1 --2- 172.21.15.39:0/2544272044 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5a300ffd20 0x7f5a300fb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:22:35.805 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.801+0000 7f5a2f7fe700 1 --2- 172.21.15.39:0/2544272044 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5a30102870 0x7f5a300fbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:22:35.805 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.801+0000 7f5a2effd700 1 -- 172.21.15.39:0/2544272044 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5a300ffd20 msgr2=0x7f5a300fb6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:22:35.805 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.801+0000 7f5a2effd700 1 --2- 172.21.15.39:0/2544272044 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5a300ffd20 0x7f5a300fb6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:22:35.806 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.801+0000 7f5a2effd700 1 -- 172.21.15.39:0/2544272044 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5a30102870 msgr2=0x7f5a300fbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:22:35.806 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.801+0000 7f5a2effd700 1 --2- 172.21.15.39:0/2544272044 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5a30102870 0x7f5a300fbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:22:35.806 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.801+0000 7f5a2effd700 1 -- 172.21.15.39:0/2544272044 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a28009940 con 0x7f5a30066110 2023-06-22T17:22:35.806 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.801+0000 7f5a2effd700 1 --2- 172.21.15.39:0/2544272044 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a30066110 0x7f5a300fd080 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5a1800ebc0 tx=0x7f5a1800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:22:35.806 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.802+0000 7f5a277fe700 1 -- 172.21.15.39:0/2544272044 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a18007cc0 con 0x7f5a30066110 2023-06-22T17:22:35.807 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.802+0000 7f5a35866700 1 -- 172.21.15.39:0/2544272044 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a30117b00 con 0x7f5a30066110 2023-06-22T17:22:35.807 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.802+0000 7f5a277fe700 1 -- 172.21.15.39:0/2544272044 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f5a1800f040 con 0x7f5a30066110 2023-06-22T17:22:35.807 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.802+0000 7f5a277fe700 1 -- 172.21.15.39:0/2544272044 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1800c940 con 0x7f5a30066110 2023-06-22T17:22:35.807 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.802+0000 7f5a35866700 1 -- 172.21.15.39:0/2544272044 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a30117ff0 con 0x7f5a30066110 2023-06-22T17:22:35.808 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.803+0000 7f5a35866700 1 -- 172.21.15.39:0/2544272044 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a300f8d40 con 0x7f5a30066110 2023-06-22T17:22:35.808 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.804+0000 7f5a277fe700 1 -- 172.21.15.39:0/2544272044 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 15) v1 ==== 129145+0+0 (secure 0 0 0) 0x7f5a1800cd30 con 0x7f5a30066110 2023-06-22T17:22:35.808 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.805+0000 7f5a277fe700 1 --2- 172.21.15.39:0/2544272044 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f5a1c09b670 0x7f5a1c09db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:22:35.808 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.805+0000 7f5a2e7fc700 1 --2- 172.21.15.39:0/2544272044 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f5a1c09b670 0x7f5a1c09db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:22:35.808 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.805+0000 7f5a277fe700 1 -- 172.21.15.39:0/2544272044 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 2847+0+0 (secure 0 0 0) 0x7f5a180c3370 con 0x7f5a30066110 2023-06-22T17:22:35.809 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.805+0000 7f5a2e7fc700 1 --2- 172.21.15.39:0/2544272044 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f5a1c09b670 0x7f5a1c09db20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5a20009b80 tx=0x7f5a2000b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:22:35.810 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:35.810+0000 7f5a277fe700 1 -- 172.21.15.39:0/2544272044 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f5a18090170 con 0x7f5a30066110 2023-06-22T17:22:36.017 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:36.016+0000 7f5a35866700 1 -- 172.21.15.39:0/2544272044 --> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a3005de70 con 0x7f5a1c09b670 2023-06-22T17:22:36.125 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:35 smithi101 ceph-mon[85507]: purged_snaps scrub starts 2023-06-22T17:22:36.125 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:35 smithi101 ceph-mon[85507]: purged_snaps scrub ok 2023-06-22T17:22:36.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:35 smithi101 ceph-mon[85507]: osd.2 [v2:172.21.15.133:6800/3632902853,v1:172.21.15.133:6801/3632902853] boot 2023-06-22T17:22:36.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:35 smithi101 ceph-mon[85507]: osdmap e18: 3 total, 3 up, 3 in 2023-06-22T17:22:36.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:35 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-06-22T17:22:36.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:35 smithi101 ceph-mon[85507]: pgmap v72: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-06-22T17:22:36.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:35 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2023-06-22T17:22:36.169 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:35 smithi133 ceph-mon[86268]: purged_snaps scrub starts 2023-06-22T17:22:36.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:35 smithi133 ceph-mon[86268]: purged_snaps scrub ok 2023-06-22T17:22:36.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:35 smithi133 ceph-mon[86268]: osd.2 [v2:172.21.15.133:6800/3632902853,v1:172.21.15.133:6801/3632902853] boot 2023-06-22T17:22:36.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:35 smithi133 ceph-mon[86268]: osdmap e18: 3 total, 3 up, 3 in 2023-06-22T17:22:36.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:35 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-06-22T17:22:36.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:35 smithi133 ceph-mon[86268]: pgmap v72: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-06-22T17:22:36.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:35 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2023-06-22T17:22:36.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:35 smithi039 ceph-mon[84805]: purged_snaps scrub starts 2023-06-22T17:22:36.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:35 smithi039 ceph-mon[84805]: purged_snaps scrub ok 2023-06-22T17:22:36.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:35 smithi039 ceph-mon[84805]: osd.2 [v2:172.21.15.133:6800/3632902853,v1:172.21.15.133:6801/3632902853] boot 2023-06-22T17:22:36.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:35 smithi039 ceph-mon[84805]: osdmap e18: 3 total, 3 up, 3 in 2023-06-22T17:22:36.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:35 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-06-22T17:22:36.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:35 smithi039 ceph-mon[84805]: pgmap v72: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-06-22T17:22:36.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:35 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2023-06-22T17:22:37.125 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:36 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2023-06-22T17:22:37.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:36 smithi101 ceph-mon[85507]: osdmap e19: 3 total, 3 up, 3 in 2023-06-22T17:22:37.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:36 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-06-22T17:22:37.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:36 smithi101 ceph-mon[85507]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:22:37.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:36 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:22:37.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:36 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:22:37.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:36 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:37.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:36 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2023-06-22T17:22:37.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:36 smithi133 ceph-mon[86268]: osdmap e19: 3 total, 3 up, 3 in 2023-06-22T17:22:37.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:36 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-06-22T17:22:37.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:36 smithi133 ceph-mon[86268]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:22:37.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:36 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:22:37.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:36 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:22:37.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:36 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:37.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:36 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2023-06-22T17:22:37.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:36 smithi039 ceph-mon[84805]: osdmap e19: 3 total, 3 up, 3 in 2023-06-22T17:22:37.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:36 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-06-22T17:22:37.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:36 smithi039 ceph-mon[84805]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:22:37.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:36 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:22:37.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:36 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:22:37.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:36 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:38.169 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:37 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-06-22T17:22:38.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:37 smithi133 ceph-mon[86268]: osdmap e20: 3 total, 3 up, 3 in 2023-06-22T17:22:38.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:37 smithi133 ceph-mon[86268]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-06-22T17:22:38.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:37 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-06-22T17:22:38.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:37 smithi039 ceph-mon[84805]: osdmap e20: 3 total, 3 up, 3 in 2023-06-22T17:22:38.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:37 smithi039 ceph-mon[84805]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-06-22T17:22:38.183 INFO:journalctl@ceph.osd.0.smithi039.stdout:Jun 22 17:22:37 smithi039 sudo[105674]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-06-22T17:22:38.184 INFO:journalctl@ceph.osd.0.smithi039.stdout:Jun 22 17:22:37 smithi039 sudo[105674]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-06-22T17:22:38.184 INFO:journalctl@ceph.osd.0.smithi039.stdout:Jun 22 17:22:37 smithi039 sudo[105674]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-06-22T17:22:38.184 INFO:journalctl@ceph.osd.0.smithi039.stdout:Jun 22 17:22:37 smithi039 sudo[105674]: pam_unix(sudo:session): session closed for user root 2023-06-22T17:22:38.375 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:37 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-06-22T17:22:38.375 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:37 smithi101 ceph-mon[85507]: osdmap e20: 3 total, 3 up, 3 in 2023-06-22T17:22:38.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:37 smithi101 ceph-mon[85507]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-06-22T17:22:38.682 INFO:journalctl@ceph.osd.0.smithi039.stdout:Jun 22 17:22:38 smithi039 sudo[105712]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-06-22T17:22:38.682 INFO:journalctl@ceph.osd.0.smithi039.stdout:Jun 22 17:22:38 smithi039 sudo[105712]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-06-22T17:22:38.682 INFO:journalctl@ceph.osd.0.smithi039.stdout:Jun 22 17:22:38 smithi039 sudo[105712]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-06-22T17:22:38.683 INFO:journalctl@ceph.osd.0.smithi039.stdout:Jun 22 17:22:38 smithi039 sudo[105712]: pam_unix(sudo:session): session closed for user root 2023-06-22T17:22:38.875 INFO:journalctl@ceph.osd.1.smithi101.stdout:Jun 22 17:22:38 smithi101 sudo[94746]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-06-22T17:22:38.875 INFO:journalctl@ceph.osd.1.smithi101.stdout:Jun 22 17:22:38 smithi101 sudo[94746]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-06-22T17:22:38.876 INFO:journalctl@ceph.osd.1.smithi101.stdout:Jun 22 17:22:38 smithi101 sudo[94746]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-06-22T17:22:38.876 INFO:journalctl@ceph.osd.1.smithi101.stdout:Jun 22 17:22:38 smithi101 sudo[94746]: pam_unix(sudo:session): session closed for user root 2023-06-22T17:22:38.876 INFO:journalctl@ceph.osd.1.smithi101.stdout:Jun 22 17:22:38 smithi101 sudo[94749]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-06-22T17:22:38.876 INFO:journalctl@ceph.osd.1.smithi101.stdout:Jun 22 17:22:38 smithi101 sudo[94749]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-06-22T17:22:38.876 INFO:journalctl@ceph.osd.1.smithi101.stdout:Jun 22 17:22:38 smithi101 sudo[94749]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-06-22T17:22:38.876 INFO:journalctl@ceph.osd.1.smithi101.stdout:Jun 22 17:22:38 smithi101 sudo[94749]: pam_unix(sudo:session): session closed for user root 2023-06-22T17:22:38.888 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:38.886+0000 7f5a277fe700 1 -- 172.21.15.39:0/2544272044 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f5a18090540 con 0x7f5a30066110 2023-06-22T17:22:39.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:38 smithi133 ceph-mon[86268]: osdmap e21: 3 total, 3 up, 3 in 2023-06-22T17:22:39.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:38 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:39.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:38 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/433163566' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b91b9111-cb8f-4c51-85c6-96c0851d57d4"}]: dispatch 2023-06-22T17:22:39.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:38 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/433163566' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b91b9111-cb8f-4c51-85c6-96c0851d57d4"}]': finished 2023-06-22T17:22:39.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:38 smithi133 ceph-mon[86268]: osdmap e22: 4 total, 3 up, 4 in 2023-06-22T17:22:39.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:38 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-22T17:22:39.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:38 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:39.172 INFO:journalctl@ceph.osd.2.smithi133.stdout:Jun 22 17:22:39 smithi133 sudo[96694]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-06-22T17:22:39.172 INFO:journalctl@ceph.osd.2.smithi133.stdout:Jun 22 17:22:39 smithi133 sudo[96694]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-06-22T17:22:39.172 INFO:journalctl@ceph.osd.2.smithi133.stdout:Jun 22 17:22:39 smithi133 sudo[96694]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-06-22T17:22:39.172 INFO:journalctl@ceph.osd.2.smithi133.stdout:Jun 22 17:22:39 smithi133 sudo[96694]: pam_unix(sudo:session): session closed for user root 2023-06-22T17:22:39.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:38 smithi039 ceph-mon[84805]: osdmap e21: 3 total, 3 up, 3 in 2023-06-22T17:22:39.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:38 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:39.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:38 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/433163566' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b91b9111-cb8f-4c51-85c6-96c0851d57d4"}]: dispatch 2023-06-22T17:22:39.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:38 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/433163566' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b91b9111-cb8f-4c51-85c6-96c0851d57d4"}]': finished 2023-06-22T17:22:39.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:38 smithi039 ceph-mon[84805]: osdmap e22: 4 total, 3 up, 4 in 2023-06-22T17:22:39.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:38 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-22T17:22:39.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:38 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:39.375 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:38 smithi101 ceph-mon[85507]: osdmap e21: 3 total, 3 up, 3 in 2023-06-22T17:22:39.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:38 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:39.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:38 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/433163566' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b91b9111-cb8f-4c51-85c6-96c0851d57d4"}]: dispatch 2023-06-22T17:22:39.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:38 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/433163566' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b91b9111-cb8f-4c51-85c6-96c0851d57d4"}]': finished 2023-06-22T17:22:39.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:38 smithi101 ceph-mon[85507]: osdmap e22: 4 total, 3 up, 4 in 2023-06-22T17:22:39.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:38 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-22T17:22:39.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:38 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:39.670 INFO:journalctl@ceph.osd.2.smithi133.stdout:Jun 22 17:22:39 smithi133 sudo[96697]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-06-22T17:22:39.670 INFO:journalctl@ceph.osd.2.smithi133.stdout:Jun 22 17:22:39 smithi133 sudo[96697]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-06-22T17:22:39.670 INFO:journalctl@ceph.osd.2.smithi133.stdout:Jun 22 17:22:39 smithi133 sudo[96697]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-06-22T17:22:39.670 INFO:journalctl@ceph.osd.2.smithi133.stdout:Jun 22 17:22:39 smithi133 sudo[96697]: pam_unix(sudo:session): session closed for user root 2023-06-22T17:22:40.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:40 smithi101 ceph-mon[85507]: mgrmap e16: x(active, since 2m), standbys: y, z 2023-06-22T17:22:40.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:40 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/2144216895' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:22:40.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:40 smithi101 ceph-mon[85507]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-06-22T17:22:40.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:40 smithi133 ceph-mon[86268]: mgrmap e16: x(active, since 2m), standbys: y, z 2023-06-22T17:22:40.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:40 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/2144216895' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:22:40.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:40 smithi133 ceph-mon[86268]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-06-22T17:22:40.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:40 smithi039 ceph-mon[84805]: mgrmap e16: x(active, since 2m), standbys: y, z 2023-06-22T17:22:40.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:40 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/2144216895' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:22:40.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:40 smithi039 ceph-mon[84805]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-06-22T17:22:41.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:40 smithi039 sudo[105804]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-06-22T17:22:41.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:40 smithi039 sudo[105804]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-06-22T17:22:41.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:40 smithi039 sudo[105804]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-06-22T17:22:41.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:40 smithi039 sudo[105804]: pam_unix(sudo:session): session closed for user root 2023-06-22T17:22:41.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:41 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:41.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:41 smithi101 ceph-mon[85507]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-06-22T17:22:41.669 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:41 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:41.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:41 smithi133 ceph-mon[86268]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-06-22T17:22:41.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:41 smithi133 sudo[96700]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-06-22T17:22:41.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:41 smithi133 sudo[96700]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-06-22T17:22:41.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:41 smithi133 sudo[96700]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-06-22T17:22:41.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:41 smithi133 sudo[96700]: pam_unix(sudo:session): session closed for user root 2023-06-22T17:22:41.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:41 smithi039 sudo[106178]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-06-22T17:22:41.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:41 smithi039 sudo[106178]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-06-22T17:22:41.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:41 smithi039 sudo[106178]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-06-22T17:22:41.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:41 smithi039 sudo[106178]: pam_unix(sudo:session): session closed for user root 2023-06-22T17:22:41.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:41 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:41.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:41 smithi039 ceph-mon[84805]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-06-22T17:22:42.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:41 smithi133 sudo[96703]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-06-22T17:22:42.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:41 smithi133 sudo[96703]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-06-22T17:22:42.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:41 smithi133 sudo[96703]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-06-22T17:22:42.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:41 smithi133 sudo[96703]: pam_unix(sudo:session): session closed for user root 2023-06-22T17:22:42.309 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:42 smithi101 sudo[94760]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-06-22T17:22:42.309 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:42 smithi101 sudo[94760]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-06-22T17:22:42.309 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:42 smithi101 sudo[94760]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-06-22T17:22:42.310 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:42 smithi101 sudo[94760]: pam_unix(sudo:session): session closed for user root 2023-06-22T17:22:42.310 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:42 smithi101 sudo[94763]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-06-22T17:22:42.310 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:42 smithi101 sudo[94763]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-06-22T17:22:42.310 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:42 smithi101 sudo[94763]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-06-22T17:22:42.310 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:42 smithi101 sudo[94763]: pam_unix(sudo:session): session closed for user root 2023-06-22T17:22:42.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:42 smithi101 ceph-mon[85507]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-06-22T17:22:42.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:42 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-22T17:22:42.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:42 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-22T17:22:42.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:42 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-22T17:22:42.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:42 smithi101 ceph-mon[85507]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2023-06-22T17:22:42.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:42 smithi101 ceph-mon[85507]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-06-22T17:22:42.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:42 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-22T17:22:42.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:42 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-22T17:22:42.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:42 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-22T17:22:42.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:42 smithi101 ceph-mon[85507]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-06-22T17:22:42.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:42 smithi101 ceph-mon[85507]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-06-22T17:22:42.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:42 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-22T17:22:42.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:42 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-22T17:22:42.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:42 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-22T17:22:42.628 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:42 smithi101 ceph-mon[85507]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-06-22T17:22:42.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:42 smithi133 ceph-mon[86268]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-06-22T17:22:42.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:42 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-22T17:22:42.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:42 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-22T17:22:42.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:42 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-22T17:22:42.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:42 smithi133 ceph-mon[86268]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2023-06-22T17:22:42.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:42 smithi133 ceph-mon[86268]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-06-22T17:22:42.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:42 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-22T17:22:42.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:42 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-22T17:22:42.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:42 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-22T17:22:42.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:42 smithi133 ceph-mon[86268]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-06-22T17:22:42.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:42 smithi133 ceph-mon[86268]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-06-22T17:22:42.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:42 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-22T17:22:42.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:42 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-22T17:22:42.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:42 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-22T17:22:42.672 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:42 smithi133 ceph-mon[86268]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-06-22T17:22:42.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:42 smithi039 ceph-mon[84805]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-06-22T17:22:42.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:42 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-22T17:22:42.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:42 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-22T17:22:42.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:42 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-22T17:22:42.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:42 smithi039 ceph-mon[84805]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2023-06-22T17:22:42.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:42 smithi039 ceph-mon[84805]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-06-22T17:22:42.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:42 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-22T17:22:42.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:42 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-22T17:22:42.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:42 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-22T17:22:42.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:42 smithi039 ceph-mon[84805]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-06-22T17:22:42.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:42 smithi039 ceph-mon[84805]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-06-22T17:22:42.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:42 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-22T17:22:42.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:42 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-22T17:22:42.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:42 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-22T17:22:42.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:42 smithi039 ceph-mon[84805]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-06-22T17:22:44.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:44 smithi101 ceph-mon[85507]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2023-06-22T17:22:44.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:44 smithi133 ceph-mon[86268]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2023-06-22T17:22:44.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:44 smithi039 ceph-mon[84805]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2023-06-22T17:22:46.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:46 smithi101 ceph-mon[85507]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2023-06-22T17:22:46.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:46 smithi133 ceph-mon[86268]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2023-06-22T17:22:46.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:46 smithi039 ceph-mon[84805]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2023-06-22T17:22:47.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:47 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-06-22T17:22:47.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:47 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:47.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:47 smithi039 ceph-mon[84805]: Deploying daemon osd.3 on smithi039 2023-06-22T17:22:47.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:47 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-06-22T17:22:47.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:47 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:47.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:47 smithi101 ceph-mon[85507]: Deploying daemon osd.3 on smithi039 2023-06-22T17:22:47.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:47 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-06-22T17:22:47.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:47 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:47.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:47 smithi133 ceph-mon[86268]: Deploying daemon osd.3 on smithi039 2023-06-22T17:22:48.658 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:48 smithi039 ceph-mon[84805]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-06-22T17:22:48.669 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:48 smithi133 ceph-mon[86268]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-06-22T17:22:48.875 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:48 smithi101 ceph-mon[85507]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-06-22T17:22:50.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:50 smithi039 ceph-mon[84805]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2023-06-22T17:22:50.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:50 smithi133 ceph-mon[86268]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2023-06-22T17:22:50.875 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:50 smithi101 ceph-mon[85507]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2023-06-22T17:22:52.375 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:52 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:52.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:52 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:52.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:52 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:22:52.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:52 smithi101 ceph-mon[85507]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2023-06-22T17:22:52.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:52 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:52.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:52 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:52.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:52 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:22:52.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:52 smithi133 ceph-mon[86268]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2023-06-22T17:22:52.432 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:52 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:52.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:52 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:52.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:52 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:22:52.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:52 smithi039 ceph-mon[84805]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2023-06-22T17:22:54.228 INFO:teuthology.orchestra.run.smithi039.stdout:Created osd(s) 3 on host 'smithi039' 2023-06-22T17:22:54.228 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:54.226+0000 7f5a277fe700 1 -- 172.21.15.39:0/2544272044 <== mgr.14150 v2:172.21.15.39:6800/886078282 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5a3005de70 con 0x7f5a1c09b670 2023-06-22T17:22:54.230 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:54.229+0000 7f5a35866700 1 -- 172.21.15.39:0/2544272044 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f5a1c09b670 msgr2=0x7f5a1c09db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:22:54.230 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:54.229+0000 7f5a35866700 1 --2- 172.21.15.39:0/2544272044 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f5a1c09b670 0x7f5a1c09db20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5a20009b80 tx=0x7f5a2000b040 comp rx=0 tx=0).stop 2023-06-22T17:22:54.230 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:54.229+0000 7f5a35866700 1 -- 172.21.15.39:0/2544272044 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a30066110 msgr2=0x7f5a300fd080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:22:54.230 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:54.229+0000 7f5a35866700 1 --2- 172.21.15.39:0/2544272044 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a30066110 0x7f5a300fd080 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5a1800ebc0 tx=0x7f5a1800b040 comp rx=0 tx=0).stop 2023-06-22T17:22:54.230 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:54.229+0000 7f5a35866700 1 -- 172.21.15.39:0/2544272044 shutdown_connections 2023-06-22T17:22:54.231 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:54.229+0000 7f5a35866700 1 --2- 172.21.15.39:0/2544272044 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f5a1c09b670 0x7f5a1c09db20 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:22:54.231 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:54.229+0000 7f5a35866700 1 --2- 172.21.15.39:0/2544272044 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5a30102870 0x7f5a300fbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:22:54.231 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:54.229+0000 7f5a35866700 1 --2- 172.21.15.39:0/2544272044 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a30066110 0x7f5a300fd080 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:22:54.231 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:54.229+0000 7f5a35866700 1 --2- 172.21.15.39:0/2544272044 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5a300ffd20 0x7f5a300fb6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:22:54.231 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:54.229+0000 7f5a35866700 1 -- 172.21.15.39:0/2544272044 >> 172.21.15.39:0/2544272044 conn(0x7f5a300f7210 msgr2=0x7f5a300fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:22:54.231 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:54.229+0000 7f5a35866700 1 -- 172.21.15.39:0/2544272044 shutdown_connections 2023-06-22T17:22:54.231 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:22:54.229+0000 7f5a35866700 1 -- 172.21.15.39:0/2544272044 wait complete. 2023-06-22T17:22:54.620 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:54 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:54.621 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:54 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:54.621 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:54 smithi039 ceph-mon[84805]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2023-06-22T17:22:54.621 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:54 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:54.621 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:54 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:22:54.621 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:54 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:54.621 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:54 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:54.622 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:54 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:54.622 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:54 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:22:54.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:54 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:54.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:54 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:54.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:54 smithi101 ceph-mon[85507]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2023-06-22T17:22:54.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:54 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:54.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:54 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:22:54.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:54 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:54.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:54 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:54.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:54 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:54.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:54 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:22:54.644 DEBUG:teuthology.orchestra.run.smithi039:osd.3> sudo journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.3.service 2023-06-22T17:22:54.647 INFO:tasks.cephadm:Deploying osd.4 on smithi101 with /dev/vg_nvme/lv_3... 2023-06-22T17:22:54.647 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-06-22T17:22:54.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:54 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:54.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:54 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:54.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:54 smithi133 ceph-mon[86268]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2023-06-22T17:22:54.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:54 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:54.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:54 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:22:54.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:54 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:54.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:54 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:54.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:54 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:54.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:54 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:22:54.677 INFO:journalctl@ceph.osd.3.smithi039.stdout:-- Logs begin at Thu 2023-06-22 17:07:05 UTC. -- 2023-06-22T17:22:55.233 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.b/config 2023-06-22T17:22:55.306 INFO:journalctl@ceph.osd.3.smithi039.stdout:Jun 22 17:22:55 smithi039 conmon[108578]: 2023-06-22T17:22:55.045+0000 7fdf14d7d540 -1 osd.3 0 log_to_monitors true 2023-06-22T17:22:55.609 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:55 smithi101 ceph-mon[85507]: from='osd.3 [v2:172.21.15.39:6810/1215819990,v1:172.21.15.39:6811/1215819990]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-06-22T17:22:55.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:55 smithi133 ceph-mon[86268]: from='osd.3 [v2:172.21.15.39:6810/1215819990,v1:172.21.15.39:6811/1215819990]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-06-22T17:22:55.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:55 smithi039 ceph-mon[84805]: from='osd.3 [v2:172.21.15.39:6810/1215819990,v1:172.21.15.39:6811/1215819990]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-06-22T17:22:56.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:56 smithi101 ceph-mon[85507]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2023-06-22T17:22:56.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:56 smithi101 ceph-mon[85507]: from='osd.3 [v2:172.21.15.39:6810/1215819990,v1:172.21.15.39:6811/1215819990]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-06-22T17:22:56.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:56 smithi101 ceph-mon[85507]: osdmap e23: 4 total, 3 up, 4 in 2023-06-22T17:22:56.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:56 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-22T17:22:56.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:56 smithi101 ceph-mon[85507]: from='osd.3 [v2:172.21.15.39:6810/1215819990,v1:172.21.15.39:6811/1215819990]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-06-22T17:22:56.631 INFO:journalctl@ceph.osd.3.smithi039.stdout:Jun 22 17:22:56 smithi039 conmon[108578]: 2023-06-22T17:22:56.323+0000 7fdf0b562700 -1 osd.3 0 waiting for initial osdmap 2023-06-22T17:22:56.631 INFO:journalctl@ceph.osd.3.smithi039.stdout:Jun 22 17:22:56 smithi039 conmon[108578]: 2023-06-22T17:22:56.332+0000 7fdf08361700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-22T17:22:56.631 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:56 smithi039 ceph-mon[84805]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2023-06-22T17:22:56.631 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:56 smithi039 ceph-mon[84805]: from='osd.3 [v2:172.21.15.39:6810/1215819990,v1:172.21.15.39:6811/1215819990]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-06-22T17:22:56.631 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:56 smithi039 ceph-mon[84805]: osdmap e23: 4 total, 3 up, 4 in 2023-06-22T17:22:56.631 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:56 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-22T17:22:56.632 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:56 smithi039 ceph-mon[84805]: from='osd.3 [v2:172.21.15.39:6810/1215819990,v1:172.21.15.39:6811/1215819990]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-06-22T17:22:56.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:56 smithi133 ceph-mon[86268]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2023-06-22T17:22:56.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:56 smithi133 ceph-mon[86268]: from='osd.3 [v2:172.21.15.39:6810/1215819990,v1:172.21.15.39:6811/1215819990]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-06-22T17:22:56.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:56 smithi133 ceph-mon[86268]: osdmap e23: 4 total, 3 up, 4 in 2023-06-22T17:22:56.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:56 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-22T17:22:56.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:56 smithi133 ceph-mon[86268]: from='osd.3 [v2:172.21.15.39:6810/1215819990,v1:172.21.15.39:6811/1215819990]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-06-22T17:22:56.850 INFO:teuthology.orchestra.run.smithi101.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-06-22T17:22:56.851 INFO:teuthology.orchestra.run.smithi101.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-06-22T17:22:56.851 INFO:teuthology.orchestra.run.smithi101.stderr: stderr: 10+0 records in 2023-06-22T17:22:56.851 INFO:teuthology.orchestra.run.smithi101.stderr:10+0 records out 2023-06-22T17:22:56.851 INFO:teuthology.orchestra.run.smithi101.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011326 s, 926 MB/s 2023-06-22T17:22:56.851 INFO:teuthology.orchestra.run.smithi101.stderr:--> Zapping successful for: 2023-06-22T17:22:57.265 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph orch daemon add osd smithi101:vg_nvme/lv_3 2023-06-22T17:22:57.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:57 smithi101 ceph-mon[85507]: from='osd.3 [v2:172.21.15.39:6810/1215819990,v1:172.21.15.39:6811/1215819990]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2023-06-22T17:22:57.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:57 smithi101 ceph-mon[85507]: osdmap e24: 4 total, 3 up, 4 in 2023-06-22T17:22:57.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:57 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-22T17:22:57.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:57 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-22T17:22:57.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:57 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:57.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:57 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:57.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:57 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:22:57.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:57 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:22:57.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:57 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:57.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:57 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:57.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:57 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:22:57.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:57 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:57.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:57 smithi133 ceph-mon[86268]: from='osd.3 [v2:172.21.15.39:6810/1215819990,v1:172.21.15.39:6811/1215819990]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2023-06-22T17:22:57.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:57 smithi133 ceph-mon[86268]: osdmap e24: 4 total, 3 up, 4 in 2023-06-22T17:22:57.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:57 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-22T17:22:57.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:57 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-22T17:22:57.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:57 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:57.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:57 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:57.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:57 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:22:57.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:57 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:22:57.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:57 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:57.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:57 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:57.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:57 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:22:57.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:57 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:57.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:57 smithi039 ceph-mon[84805]: from='osd.3 [v2:172.21.15.39:6810/1215819990,v1:172.21.15.39:6811/1215819990]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2023-06-22T17:22:57.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:57 smithi039 ceph-mon[84805]: osdmap e24: 4 total, 3 up, 4 in 2023-06-22T17:22:57.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:57 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-22T17:22:57.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:57 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-22T17:22:57.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:57 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:57.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:57 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:57.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:57 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:22:57.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:57 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:22:57.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:57 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:57.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:57 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:57.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:57 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:22:57.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:57 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:22:57.876 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.b/config 2023-06-22T17:22:58.375 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:58 smithi101 ceph-mon[85507]: purged_snaps scrub starts 2023-06-22T17:22:58.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:58 smithi101 ceph-mon[85507]: purged_snaps scrub ok 2023-06-22T17:22:58.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:58 smithi101 ceph-mon[85507]: Detected new or changed devices on smithi039 2023-06-22T17:22:58.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:58 smithi101 ceph-mon[85507]: Adjusting osd_memory_target on smithi039 to 8584M 2023-06-22T17:22:58.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:58 smithi101 ceph-mon[85507]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2023-06-22T17:22:58.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:58 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-22T17:22:58.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:58 smithi101 ceph-mon[85507]: osd.3 [v2:172.21.15.39:6810/1215819990,v1:172.21.15.39:6811/1215819990] boot 2023-06-22T17:22:58.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:58 smithi101 ceph-mon[85507]: osdmap e25: 4 total, 4 up, 4 in 2023-06-22T17:22:58.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:58 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-22T17:22:58.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:58 smithi133 ceph-mon[86268]: purged_snaps scrub starts 2023-06-22T17:22:58.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:58 smithi133 ceph-mon[86268]: purged_snaps scrub ok 2023-06-22T17:22:58.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:58 smithi133 ceph-mon[86268]: Detected new or changed devices on smithi039 2023-06-22T17:22:58.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:58 smithi133 ceph-mon[86268]: Adjusting osd_memory_target on smithi039 to 8584M 2023-06-22T17:22:58.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:58 smithi133 ceph-mon[86268]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2023-06-22T17:22:58.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:58 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-22T17:22:58.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:58 smithi133 ceph-mon[86268]: osd.3 [v2:172.21.15.39:6810/1215819990,v1:172.21.15.39:6811/1215819990] boot 2023-06-22T17:22:58.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:58 smithi133 ceph-mon[86268]: osdmap e25: 4 total, 4 up, 4 in 2023-06-22T17:22:58.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:58 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-22T17:22:58.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:58 smithi039 ceph-mon[84805]: purged_snaps scrub starts 2023-06-22T17:22:58.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:58 smithi039 ceph-mon[84805]: purged_snaps scrub ok 2023-06-22T17:22:58.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:58 smithi039 ceph-mon[84805]: Detected new or changed devices on smithi039 2023-06-22T17:22:58.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:58 smithi039 ceph-mon[84805]: Adjusting osd_memory_target on smithi039 to 8584M 2023-06-22T17:22:58.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:58 smithi039 ceph-mon[84805]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2023-06-22T17:22:58.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:58 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-22T17:22:58.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:58 smithi039 ceph-mon[84805]: osd.3 [v2:172.21.15.39:6810/1215819990,v1:172.21.15.39:6811/1215819990] boot 2023-06-22T17:22:58.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:58 smithi039 ceph-mon[84805]: osdmap e25: 4 total, 4 up, 4 in 2023-06-22T17:22:58.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:58 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-22T17:22:59.110 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.108+0000 7fb880b85700 1 -- 172.21.15.101:0/3000199666 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb87c0fcd30 msgr2=0x7fb87c0fd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:22:59.110 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.108+0000 7fb880b85700 1 --2- 172.21.15.101:0/3000199666 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb87c0fcd30 0x7fb87c0fd180 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb8700049c0 tx=0x7fb870009e30 comp rx=0 tx=0).stop 2023-06-22T17:22:59.110 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.108+0000 7fb880b85700 1 -- 172.21.15.101:0/3000199666 shutdown_connections 2023-06-22T17:22:59.110 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.108+0000 7fb880b85700 1 --2- 172.21.15.101:0/3000199666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb87c0fd6c0 0x7fb87c103f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:22:59.110 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.108+0000 7fb880b85700 1 --2- 172.21.15.101:0/3000199666 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb87c0fbb20 0x7fb87c0fbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:22:59.111 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.108+0000 7fb880b85700 1 --2- 172.21.15.101:0/3000199666 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb87c0fcd30 0x7fb87c0fd180 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:22:59.111 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.108+0000 7fb880b85700 1 -- 172.21.15.101:0/3000199666 >> 172.21.15.101:0/3000199666 conn(0x7fb87c0f7260 msgr2=0x7fb87c0f96b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:22:59.111 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.108+0000 7fb880b85700 1 -- 172.21.15.101:0/3000199666 shutdown_connections 2023-06-22T17:22:59.111 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.108+0000 7fb880b85700 1 -- 172.21.15.101:0/3000199666 wait complete. 2023-06-22T17:22:59.111 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.108+0000 7fb880b85700 1 Processor -- start 2023-06-22T17:22:59.111 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.108+0000 7fb880b85700 1 -- start start 2023-06-22T17:22:59.111 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.109+0000 7fb880b85700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb87c0fbb20 0x7fb87c10d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:22:59.111 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.109+0000 7fb880b85700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb87c0fcd30 0x7fb87c10dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:22:59.112 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.109+0000 7fb880b85700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb87c0fd6c0 0x7fb87c114f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:22:59.112 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.109+0000 7fb880b85700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c115540 con 0x7fb87c0fd6c0 2023-06-22T17:22:59.112 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.109+0000 7fb880b85700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c1156b0 con 0x7fb87c0fcd30 2023-06-22T17:22:59.112 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.109+0000 7fb880b85700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c115870 con 0x7fb87c0fbb20 2023-06-22T17:22:59.112 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.109+0000 7fb879d9b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb87c0fcd30 0x7fb87c10dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:22:59.112 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.109+0000 7fb879d9b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb87c0fcd30 0x7fb87c10dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:55984/0 (socket says 172.21.15.101:55984) 2023-06-22T17:22:59.112 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.109+0000 7fb879d9b700 1 -- 172.21.15.101:0/2073830399 learned_addr learned my addr 172.21.15.101:0/2073830399 (peer_addr_for_me v2:172.21.15.101:0/0) 2023-06-22T17:22:59.113 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.109+0000 7fb87a59c700 1 --2- 172.21.15.101:0/2073830399 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb87c0fbb20 0x7fb87c10d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:22:59.113 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.109+0000 7fb87ad9d700 1 --2- 172.21.15.101:0/2073830399 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb87c0fd6c0 0x7fb87c114f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:22:59.113 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.109+0000 7fb879d9b700 1 -- 172.21.15.101:0/2073830399 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb87c0fbb20 msgr2=0x7fb87c10d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:22:59.113 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.109+0000 7fb879d9b700 1 --2- 172.21.15.101:0/2073830399 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb87c0fbb20 0x7fb87c10d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:22:59.113 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.109+0000 7fb879d9b700 1 -- 172.21.15.101:0/2073830399 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb87c0fd6c0 msgr2=0x7fb87c114f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:22:59.113 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.109+0000 7fb879d9b700 1 --2- 172.21.15.101:0/2073830399 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb87c0fd6c0 0x7fb87c114f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:22:59.113 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.109+0000 7fb879d9b700 1 -- 172.21.15.101:0/2073830399 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb870005040 con 0x7fb87c0fcd30 2023-06-22T17:22:59.114 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.109+0000 7fb879d9b700 1 --2- 172.21.15.101:0/2073830399 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb87c0fcd30 0x7fb87c10dea0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb87000bf20 tx=0x7fb870009e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:22:59.114 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.109+0000 7fb86affd700 1 -- 172.21.15.101:0/2073830399 <== mon.2 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87000e040 con 0x7fb87c0fcd30 2023-06-22T17:22:59.114 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.109+0000 7fb86affd700 1 -- 172.21.15.101:0/2073830399 <== mon.2 v2:172.21.15.101:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7fb870009830 con 0x7fb87c0fcd30 2023-06-22T17:22:59.114 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.109+0000 7fb86affd700 1 -- 172.21.15.101:0/2073830399 <== mon.2 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87001c3f0 con 0x7fb87c0fcd30 2023-06-22T17:22:59.114 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.109+0000 7fb880b85700 1 -- 172.21.15.101:0/2073830399 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb87c115af0 con 0x7fb87c0fcd30 2023-06-22T17:22:59.114 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.109+0000 7fb880b85700 1 -- 172.21.15.101:0/2073830399 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb87c115fd0 con 0x7fb87c0fcd30 2023-06-22T17:22:59.115 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.110+0000 7fb880b85700 1 -- 172.21.15.101:0/2073830399 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb87c1085f0 con 0x7fb87c0fcd30 2023-06-22T17:22:59.115 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.111+0000 7fb86affd700 1 -- 172.21.15.101:0/2073830399 <== mon.2 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7fb870009350 con 0x7fb87c0fcd30 2023-06-22T17:22:59.115 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.112+0000 7fb86affd700 1 --2- 172.21.15.101:0/2073830399 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fb86409b780 0x7fb86409dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:22:59.115 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.112+0000 7fb86affd700 1 -- 172.21.15.101:0/2073830399 <== mon.2 v2:172.21.15.101:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 3694+0+0 (secure 0 0 0) 0x7fb8700c4f30 con 0x7fb87c0fcd30 2023-06-22T17:22:59.115 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.112+0000 7fb87a59c700 1 --2- 172.21.15.101:0/2073830399 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fb86409b780 0x7fb86409dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:22:59.116 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.113+0000 7fb87a59c700 1 --2- 172.21.15.101:0/2073830399 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fb86409b780 0x7fb86409dc30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb86c009fd0 tx=0x7fb86c00b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:22:59.116 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.116+0000 7fb86affd700 1 -- 172.21.15.101:0/2073830399 <== mon.2 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fb870091900 con 0x7fb87c0fcd30 2023-06-22T17:22:59.305 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:22:59.305+0000 7fb880b85700 1 -- 172.21.15.101:0/2073830399 --> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fb87c05de70 con 0x7fb86409b780 2023-06-22T17:22:59.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:59 smithi101 ceph-mon[85507]: osdmap e26: 4 total, 4 up, 4 in 2023-06-22T17:22:59.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:59 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:22:59.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:59 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:22:59.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:22:59 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:59.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:59 smithi133 ceph-mon[86268]: osdmap e26: 4 total, 4 up, 4 in 2023-06-22T17:22:59.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:59 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:22:59.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:59 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:22:59.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:22:59 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:22:59.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:59 smithi039 ceph-mon[84805]: osdmap e26: 4 total, 4 up, 4 in 2023-06-22T17:22:59.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:59 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:22:59.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:59 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:22:59.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:22:59 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:23:00.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:00 smithi101 ceph-mon[85507]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:23:00.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:00 smithi101 ceph-mon[85507]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-06-22T17:23:00.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:00 smithi101 ceph-mon[85507]: osdmap e27: 4 total, 4 up, 4 in 2023-06-22T17:23:00.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:00 smithi133 ceph-mon[86268]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:23:00.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:00 smithi133 ceph-mon[86268]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-06-22T17:23:00.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:00 smithi133 ceph-mon[86268]: osdmap e27: 4 total, 4 up, 4 in 2023-06-22T17:23:00.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:00 smithi039 ceph-mon[84805]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:23:00.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:00 smithi039 ceph-mon[84805]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-06-22T17:23:00.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:00 smithi039 ceph-mon[84805]: osdmap e27: 4 total, 4 up, 4 in 2023-06-22T17:23:01.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:01 smithi101 ceph-mon[85507]: osdmap e28: 4 total, 4 up, 4 in 2023-06-22T17:23:01.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:01 smithi101 ceph-mon[85507]: osdmap e29: 4 total, 4 up, 4 in 2023-06-22T17:23:01.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:01 smithi133 ceph-mon[86268]: osdmap e28: 4 total, 4 up, 4 in 2023-06-22T17:23:01.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:01 smithi133 ceph-mon[86268]: osdmap e29: 4 total, 4 up, 4 in 2023-06-22T17:23:01.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:01 smithi039 ceph-mon[84805]: osdmap e28: 4 total, 4 up, 4 in 2023-06-22T17:23:01.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:01 smithi039 ceph-mon[84805]: osdmap e29: 4 total, 4 up, 4 in 2023-06-22T17:23:02.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:02 smithi133 ceph-mon[86268]: pgmap v95: 1 pgs: 1 remapped+peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-06-22T17:23:02.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:02 smithi133 ceph-mon[86268]: from='client.? 172.21.15.101:0/3625854496' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14f6d038-1e3c-41e0-9be6-3615099ac5fc"}]: dispatch 2023-06-22T17:23:02.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:02 smithi133 ceph-mon[86268]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14f6d038-1e3c-41e0-9be6-3615099ac5fc"}]: dispatch 2023-06-22T17:23:02.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:02 smithi133 ceph-mon[86268]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14f6d038-1e3c-41e0-9be6-3615099ac5fc"}]': finished 2023-06-22T17:23:02.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:02 smithi133 ceph-mon[86268]: osdmap e30: 5 total, 4 up, 5 in 2023-06-22T17:23:02.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-06-22T17:23:02.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:02 smithi133 ceph-mon[86268]: from='client.? 172.21.15.101:0/381168115' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:23:02.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:02 smithi039 ceph-mon[84805]: pgmap v95: 1 pgs: 1 remapped+peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-06-22T17:23:02.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:02 smithi039 ceph-mon[84805]: from='client.? 172.21.15.101:0/3625854496' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14f6d038-1e3c-41e0-9be6-3615099ac5fc"}]: dispatch 2023-06-22T17:23:02.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:02 smithi039 ceph-mon[84805]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14f6d038-1e3c-41e0-9be6-3615099ac5fc"}]: dispatch 2023-06-22T17:23:02.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:02 smithi039 ceph-mon[84805]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14f6d038-1e3c-41e0-9be6-3615099ac5fc"}]': finished 2023-06-22T17:23:02.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:02 smithi039 ceph-mon[84805]: osdmap e30: 5 total, 4 up, 5 in 2023-06-22T17:23:02.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:02 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-06-22T17:23:02.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:02 smithi039 ceph-mon[84805]: from='client.? 172.21.15.101:0/381168115' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:23:02.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:02 smithi101 ceph-mon[85507]: pgmap v95: 1 pgs: 1 remapped+peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-06-22T17:23:02.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:02 smithi101 ceph-mon[85507]: from='client.? 172.21.15.101:0/3625854496' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14f6d038-1e3c-41e0-9be6-3615099ac5fc"}]: dispatch 2023-06-22T17:23:02.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:02 smithi101 ceph-mon[85507]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14f6d038-1e3c-41e0-9be6-3615099ac5fc"}]: dispatch 2023-06-22T17:23:02.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:02 smithi101 ceph-mon[85507]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14f6d038-1e3c-41e0-9be6-3615099ac5fc"}]': finished 2023-06-22T17:23:02.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:02 smithi101 ceph-mon[85507]: osdmap e30: 5 total, 4 up, 5 in 2023-06-22T17:23:02.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:02 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-06-22T17:23:02.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:02 smithi101 ceph-mon[85507]: from='client.? 172.21.15.101:0/381168115' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:23:04.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:04 smithi133 ceph-mon[86268]: pgmap v98: 1 pgs: 1 remapped+peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-06-22T17:23:04.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:04 smithi039 ceph-mon[84805]: pgmap v98: 1 pgs: 1 remapped+peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-06-22T17:23:04.875 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:04 smithi101 ceph-mon[85507]: pgmap v98: 1 pgs: 1 remapped+peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-06-22T17:23:06.667 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:06 smithi101 ceph-mon[85507]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 25 MiB used, 358 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2023-06-22T17:23:06.669 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:06 smithi133 ceph-mon[86268]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 25 MiB used, 358 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2023-06-22T17:23:06.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:06 smithi039 ceph-mon[84805]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 25 MiB used, 358 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2023-06-22T17:23:08.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:08 smithi101 ceph-mon[85507]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 83 KiB/s, 0 objects/s recovering 2023-06-22T17:23:08.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:08 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-06-22T17:23:08.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:08 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:23:08.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:08 smithi133 ceph-mon[86268]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 83 KiB/s, 0 objects/s recovering 2023-06-22T17:23:08.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:08 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-06-22T17:23:08.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:08 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:23:08.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:08 smithi039 ceph-mon[84805]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 83 KiB/s, 0 objects/s recovering 2023-06-22T17:23:08.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:08 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-06-22T17:23:08.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:08 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:23:09.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:09 smithi101 ceph-mon[85507]: Deploying daemon osd.4 on smithi101 2023-06-22T17:23:09.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:09 smithi133 ceph-mon[86268]: Deploying daemon osd.4 on smithi101 2023-06-22T17:23:09.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:09 smithi039 ceph-mon[84805]: Deploying daemon osd.4 on smithi101 2023-06-22T17:23:10.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:10 smithi101 ceph-mon[85507]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-06-22T17:23:10.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:10 smithi133 ceph-mon[86268]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-06-22T17:23:10.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:10 smithi039 ceph-mon[84805]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-06-22T17:23:12.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:12 smithi101 ceph-mon[85507]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-06-22T17:23:12.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:12 smithi133 ceph-mon[86268]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-06-22T17:23:12.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:12 smithi039 ceph-mon[84805]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-06-22T17:23:13.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:13 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:13.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:13 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:13.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:13 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:23:13.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:13 smithi101 ceph-mon[85507]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 49 KiB/s, 0 objects/s recovering 2023-06-22T17:23:13.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:13 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:13.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:13 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:13.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:13 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:23:13.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:13 smithi133 ceph-mon[86268]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 49 KiB/s, 0 objects/s recovering 2023-06-22T17:23:13.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:13 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:13.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:13 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:13.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:13 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:23:13.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:13 smithi039 ceph-mon[84805]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 49 KiB/s, 0 objects/s recovering 2023-06-22T17:23:15.216 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:23:15.215+0000 7fb86affd700 1 -- 172.21.15.101:0/2073830399 <== mgr.14150 v2:172.21.15.39:6800/886078282 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb87c05de70 con 0x7fb86409b780 2023-06-22T17:23:15.217 INFO:teuthology.orchestra.run.smithi101.stdout:Created osd(s) 4 on host 'smithi101' 2023-06-22T17:23:15.219 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:23:15.217+0000 7fb880b85700 1 -- 172.21.15.101:0/2073830399 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fb86409b780 msgr2=0x7fb86409dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:23:15.219 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:23:15.217+0000 7fb880b85700 1 --2- 172.21.15.101:0/2073830399 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fb86409b780 0x7fb86409dc30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb86c009fd0 tx=0x7fb86c00b040 comp rx=0 tx=0).stop 2023-06-22T17:23:15.219 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:23:15.217+0000 7fb880b85700 1 -- 172.21.15.101:0/2073830399 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb87c0fcd30 msgr2=0x7fb87c10dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:23:15.219 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:23:15.217+0000 7fb880b85700 1 --2- 172.21.15.101:0/2073830399 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb87c0fcd30 0x7fb87c10dea0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb87000bf20 tx=0x7fb870009e30 comp rx=0 tx=0).stop 2023-06-22T17:23:15.219 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:23:15.217+0000 7fb880b85700 1 -- 172.21.15.101:0/2073830399 shutdown_connections 2023-06-22T17:23:15.219 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:23:15.217+0000 7fb880b85700 1 --2- 172.21.15.101:0/2073830399 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fb86409b780 0x7fb86409dc30 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:23:15.220 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:23:15.217+0000 7fb880b85700 1 --2- 172.21.15.101:0/2073830399 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb87c0fd6c0 0x7fb87c114f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:23:15.220 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:23:15.217+0000 7fb880b85700 1 --2- 172.21.15.101:0/2073830399 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb87c0fbb20 0x7fb87c10d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:23:15.220 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:23:15.217+0000 7fb880b85700 1 --2- 172.21.15.101:0/2073830399 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb87c0fcd30 0x7fb87c10dea0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:23:15.220 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:23:15.217+0000 7fb880b85700 1 -- 172.21.15.101:0/2073830399 >> 172.21.15.101:0/2073830399 conn(0x7fb87c0f7260 msgr2=0x7fb87c0f9400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:23:15.220 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:23:15.217+0000 7fb880b85700 1 -- 172.21.15.101:0/2073830399 shutdown_connections 2023-06-22T17:23:15.220 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:23:15.217+0000 7fb880b85700 1 -- 172.21.15.101:0/2073830399 wait complete. 2023-06-22T17:23:15.626 DEBUG:teuthology.orchestra.run.smithi101:osd.4> sudo journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.4.service 2023-06-22T17:23:15.629 INFO:tasks.cephadm:Deploying osd.5 on smithi133 with /dev/vg_nvme/lv_3... 2023-06-22T17:23:15.629 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-06-22T17:23:15.656 INFO:journalctl@ceph.osd.4.smithi101.stdout:-- Logs begin at Thu 2023-06-22 17:06:34 UTC. -- 2023-06-22T17:23:15.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:15 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:15.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:15 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:15.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:15 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:23:15.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:15 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:23:15.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:15 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:15.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:15 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:15.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:15 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:15.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:15 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:23:15.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:15 smithi101 ceph-mon[85507]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-06-22T17:23:15.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:15 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:15.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:15 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:15.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:15 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:23:15.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:15 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:23:15.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:15 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:15.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:15 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:15.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:15 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:15.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:15 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:23:15.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:15 smithi133 ceph-mon[86268]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-06-22T17:23:15.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:15 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:15.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:15 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:15.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:15 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:23:15.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:15 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:23:15.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:15 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:15.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:15 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:15.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:15 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:15.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:15 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:23:15.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:15 smithi039 ceph-mon[84805]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-06-22T17:23:16.217 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.c/config 2023-06-22T17:23:16.750 INFO:journalctl@ceph.osd.4.smithi101.stdout:Jun 22 17:23:16 smithi101 conmon[98978]: 2023-06-22T17:23:16.490+0000 7f55b1745540 -1 osd.4 0 log_to_monitors true 2023-06-22T17:23:17.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:17 smithi101 ceph-mon[85507]: from='osd.4 [v2:172.21.15.101:6808/1371317064,v1:172.21.15.101:6809/1371317064]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-06-22T17:23:17.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:17 smithi101 ceph-mon[85507]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-06-22T17:23:17.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:17 smithi133 ceph-mon[86268]: from='osd.4 [v2:172.21.15.101:6808/1371317064,v1:172.21.15.101:6809/1371317064]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-06-22T17:23:17.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:17 smithi133 ceph-mon[86268]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-06-22T17:23:17.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:17 smithi039 ceph-mon[84805]: from='osd.4 [v2:172.21.15.101:6808/1371317064,v1:172.21.15.101:6809/1371317064]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-06-22T17:23:17.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:17 smithi039 ceph-mon[84805]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-06-22T17:23:17.722 INFO:teuthology.orchestra.run.smithi133.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-06-22T17:23:17.722 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-06-22T17:23:17.722 INFO:teuthology.orchestra.run.smithi133.stderr: stderr: 10+0 records in 2023-06-22T17:23:17.722 INFO:teuthology.orchestra.run.smithi133.stderr:10+0 records out 2023-06-22T17:23:17.722 INFO:teuthology.orchestra.run.smithi133.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112339 s, 933 MB/s 2023-06-22T17:23:17.722 INFO:teuthology.orchestra.run.smithi133.stderr:--> Zapping successful for: 2023-06-22T17:23:18.160 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph orch daemon add osd smithi133:vg_nvme/lv_3 2023-06-22T17:23:18.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:18 smithi133 ceph-mon[86268]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-06-22T17:23:18.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:18 smithi133 ceph-mon[86268]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-06-22T17:23:18.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:18 smithi133 ceph-mon[86268]: osdmap e31: 5 total, 4 up, 5 in 2023-06-22T17:23:18.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:18 smithi133 ceph-mon[86268]: from='osd.4 [v2:172.21.15.101:6808/1371317064,v1:172.21.15.101:6809/1371317064]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-06-22T17:23:18.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:18 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-06-22T17:23:18.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:18 smithi133 ceph-mon[86268]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-06-22T17:23:18.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:18 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:18.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:18 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:18.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:18 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:23:18.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:18 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:23:18.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:18 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:18.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:18 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:23:18.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:18 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:23:18.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:18 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:18.625 INFO:journalctl@ceph.osd.4.smithi101.stdout:Jun 22 17:23:18 smithi101 conmon[98978]: 2023-06-22T17:23:18.335+0000 7f55a7f2a700 -1 osd.4 0 waiting for initial osdmap 2023-06-22T17:23:18.626 INFO:journalctl@ceph.osd.4.smithi101.stdout:Jun 22 17:23:18 smithi101 conmon[98978]: 2023-06-22T17:23:18.344+0000 7f55a1d23700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-22T17:23:18.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:18 smithi101 ceph-mon[85507]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-06-22T17:23:18.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:18 smithi101 ceph-mon[85507]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-06-22T17:23:18.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:18 smithi101 ceph-mon[85507]: osdmap e31: 5 total, 4 up, 5 in 2023-06-22T17:23:18.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:18 smithi101 ceph-mon[85507]: from='osd.4 [v2:172.21.15.101:6808/1371317064,v1:172.21.15.101:6809/1371317064]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-06-22T17:23:18.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:18 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-06-22T17:23:18.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:18 smithi101 ceph-mon[85507]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-06-22T17:23:18.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:18 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:18.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:18 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:18.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:18 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:23:18.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:18 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:23:18.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:18 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:18.628 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:18 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:23:18.628 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:18 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:23:18.628 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:18 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:18.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:18 smithi039 ceph-mon[84805]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-06-22T17:23:18.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:18 smithi039 ceph-mon[84805]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-06-22T17:23:18.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:18 smithi039 ceph-mon[84805]: osdmap e31: 5 total, 4 up, 5 in 2023-06-22T17:23:18.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:18 smithi039 ceph-mon[84805]: from='osd.4 [v2:172.21.15.101:6808/1371317064,v1:172.21.15.101:6809/1371317064]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-06-22T17:23:18.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:18 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-06-22T17:23:18.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:18 smithi039 ceph-mon[84805]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-06-22T17:23:18.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:18 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:18.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:18 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:18.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:18 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:23:18.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:18 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:23:18.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:18 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:18.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:18 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:23:18.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:18 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:23:18.685 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:18 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:18.745 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.c/config 2023-06-22T17:23:19.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:19 smithi101 ceph-mon[85507]: Detected new or changed devices on smithi101 2023-06-22T17:23:19.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:19 smithi101 ceph-mon[85507]: Adjusting osd_memory_target on smithi101 to 8584M 2023-06-22T17:23:19.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:19 smithi101 ceph-mon[85507]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2023-06-22T17:23:19.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:19 smithi101 ceph-mon[85507]: osdmap e32: 5 total, 4 up, 5 in 2023-06-22T17:23:19.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:19 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-06-22T17:23:19.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:19 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-06-22T17:23:19.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:19 smithi133 ceph-mon[86268]: Detected new or changed devices on smithi101 2023-06-22T17:23:19.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:19 smithi133 ceph-mon[86268]: Adjusting osd_memory_target on smithi101 to 8584M 2023-06-22T17:23:19.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:19 smithi133 ceph-mon[86268]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2023-06-22T17:23:19.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:19 smithi133 ceph-mon[86268]: osdmap e32: 5 total, 4 up, 5 in 2023-06-22T17:23:19.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:19 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-06-22T17:23:19.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:19 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-06-22T17:23:19.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:19 smithi039 ceph-mon[84805]: Detected new or changed devices on smithi101 2023-06-22T17:23:19.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:19 smithi039 ceph-mon[84805]: Adjusting osd_memory_target on smithi101 to 8584M 2023-06-22T17:23:19.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:19 smithi039 ceph-mon[84805]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2023-06-22T17:23:19.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:19 smithi039 ceph-mon[84805]: osdmap e32: 5 total, 4 up, 5 in 2023-06-22T17:23:19.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:19 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-06-22T17:23:19.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:19 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-06-22T17:23:19.971 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.969+0000 7ffad629e700 1 -- 172.21.15.133:0/2233219462 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffad00fc760 msgr2=0x7ffad0104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:23:19.971 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.969+0000 7ffad629e700 1 --2- 172.21.15.133:0/2233219462 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffad00fc760 0x7ffad0104010 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ffac8009d90 tx=0x7ffac8004d10 comp rx=0 tx=0).stop 2023-06-22T17:23:19.972 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.969+0000 7ffad629e700 1 -- 172.21.15.133:0/2233219462 shutdown_connections 2023-06-22T17:23:19.972 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.969+0000 7ffad629e700 1 --2- 172.21.15.133:0/2233219462 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffad00fc760 0x7ffad0104010 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:23:19.972 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.969+0000 7ffad629e700 1 --2- 172.21.15.133:0/2233219462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffad00fd140 0x7ffad00fd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:23:19.972 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.969+0000 7ffad629e700 1 --2- 172.21.15.133:0/2233219462 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ffad00fbdb0 0x7ffad00fc220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:23:19.972 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.969+0000 7ffad629e700 1 -- 172.21.15.133:0/2233219462 >> 172.21.15.133:0/2233219462 conn(0x7ffad00f6a50 msgr2=0x7ffad00f8e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:23:19.972 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.969+0000 7ffad629e700 1 -- 172.21.15.133:0/2233219462 shutdown_connections 2023-06-22T17:23:19.972 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.969+0000 7ffad629e700 1 -- 172.21.15.133:0/2233219462 wait complete. 2023-06-22T17:23:19.973 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.970+0000 7ffad629e700 1 Processor -- start 2023-06-22T17:23:19.974 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.970+0000 7ffad629e700 1 -- start start 2023-06-22T17:23:19.974 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.970+0000 7ffad629e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffad00fbdb0 0x7ffad010f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:23:19.974 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.970+0000 7ffad629e700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ffad00fc760 0x7ffad010fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:23:19.975 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.970+0000 7ffad629e700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffad00fd140 0x7ffad0116e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:23:19.975 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.970+0000 7ffad629e700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad0117490 con 0x7ffad00fbdb0 2023-06-22T17:23:19.975 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.970+0000 7ffad629e700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad0117600 con 0x7ffad00fc760 2023-06-22T17:23:19.975 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.970+0000 7ffad629e700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad01177c0 con 0x7ffad00fd140 2023-06-22T17:23:19.975 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.970+0000 7ffad483b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffad00fd140 0x7ffad0116e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:23:19.975 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.970+0000 7ffad483b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffad00fd140 0x7ffad0116e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:59172/0 (socket says 172.21.15.133:59172) 2023-06-22T17:23:19.975 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.970+0000 7ffacffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffad00fbdb0 0x7ffad010f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:23:19.975 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.970+0000 7ffad483b700 1 -- 172.21.15.133:0/3351525205 learned_addr learned my addr 172.21.15.133:0/3351525205 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-06-22T17:23:19.976 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.971+0000 7ffacf7fe700 1 --2- 172.21.15.133:0/3351525205 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ffad00fc760 0x7ffad010fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:23:19.976 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.971+0000 7ffad483b700 1 -- 172.21.15.133:0/3351525205 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ffad00fc760 msgr2=0x7ffad010fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:23:19.976 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.971+0000 7ffad483b700 1 --2- 172.21.15.133:0/3351525205 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ffad00fc760 0x7ffad010fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:23:19.976 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.971+0000 7ffad483b700 1 -- 172.21.15.133:0/3351525205 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffad00fbdb0 msgr2=0x7ffad010f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:23:19.976 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.971+0000 7ffad483b700 1 --2- 172.21.15.133:0/3351525205 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffad00fbdb0 0x7ffad010f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:23:19.976 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.971+0000 7ffad483b700 1 -- 172.21.15.133:0/3351525205 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac8009940 con 0x7ffad00fd140 2023-06-22T17:23:19.976 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.971+0000 7ffad483b700 1 --2- 172.21.15.133:0/3351525205 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffad00fd140 0x7ffad0116e80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ffac8009720 tx=0x7ffac8004d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:23:19.977 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.971+0000 7ffaccff9700 1 -- 172.21.15.133:0/3351525205 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac800e040 con 0x7ffad00fd140 2023-06-22T17:23:19.977 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.971+0000 7ffaccff9700 1 -- 172.21.15.133:0/3351525205 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7ffac8012d10 con 0x7ffad00fd140 2023-06-22T17:23:19.977 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.971+0000 7ffad629e700 1 -- 172.21.15.133:0/3351525205 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffad0117a40 con 0x7ffad00fd140 2023-06-22T17:23:19.977 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.971+0000 7ffaccff9700 1 -- 172.21.15.133:0/3351525205 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac801c9d0 con 0x7ffad00fd140 2023-06-22T17:23:19.977 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.971+0000 7ffad629e700 1 -- 172.21.15.133:0/3351525205 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffad0117ed0 con 0x7ffad00fd140 2023-06-22T17:23:19.977 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.972+0000 7ffad629e700 1 -- 172.21.15.133:0/3351525205 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffad0101750 con 0x7ffad00fd140 2023-06-22T17:23:19.977 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.973+0000 7ffaccff9700 1 -- 172.21.15.133:0/3351525205 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7ffac8019050 con 0x7ffad00fd140 2023-06-22T17:23:19.978 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.973+0000 7ffaccff9700 1 --2- 172.21.15.133:0/3351525205 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7ffab809b680 0x7ffab809db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:23:19.978 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.974+0000 7ffaccff9700 1 -- 172.21.15.133:0/3351525205 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 4097+0+0 (secure 0 0 0) 0x7ffac80c6290 con 0x7ffad00fd140 2023-06-22T17:23:19.978 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.974+0000 7ffacffff700 1 --2- 172.21.15.133:0/3351525205 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7ffab809b680 0x7ffab809db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:23:19.978 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.974+0000 7ffacffff700 1 --2- 172.21.15.133:0/3351525205 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7ffab809b680 0x7ffab809db30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ffad0065a30 tx=0x7ffac000b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:23:19.979 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:19.978+0000 7ffaccff9700 1 -- 172.21.15.133:0/3351525205 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7ffac8092b60 con 0x7ffad00fd140 2023-06-22T17:23:20.170 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:20.168+0000 7ffad629e700 1 -- 172.21.15.133:0/3351525205 --> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7ffad002b1b0 con 0x7ffab809b680 2023-06-22T17:23:20.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:20 smithi101 ceph-mon[85507]: purged_snaps scrub starts 2023-06-22T17:23:20.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:20 smithi101 ceph-mon[85507]: purged_snaps scrub ok 2023-06-22T17:23:20.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:20 smithi101 ceph-mon[85507]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-06-22T17:23:20.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:20 smithi101 ceph-mon[85507]: osd.4 [v2:172.21.15.101:6808/1371317064,v1:172.21.15.101:6809/1371317064] boot 2023-06-22T17:23:20.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:20 smithi101 ceph-mon[85507]: osdmap e33: 5 total, 5 up, 5 in 2023-06-22T17:23:20.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:20 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-06-22T17:23:20.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:20 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:23:20.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:20 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:23:20.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:20 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:23:20.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:20 smithi133 ceph-mon[86268]: purged_snaps scrub starts 2023-06-22T17:23:20.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:20 smithi133 ceph-mon[86268]: purged_snaps scrub ok 2023-06-22T17:23:20.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:20 smithi133 ceph-mon[86268]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-06-22T17:23:20.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:20 smithi133 ceph-mon[86268]: osd.4 [v2:172.21.15.101:6808/1371317064,v1:172.21.15.101:6809/1371317064] boot 2023-06-22T17:23:20.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:20 smithi133 ceph-mon[86268]: osdmap e33: 5 total, 5 up, 5 in 2023-06-22T17:23:20.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:20 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-06-22T17:23:20.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:20 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:23:20.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:20 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:23:20.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:20 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:23:20.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:20 smithi039 ceph-mon[84805]: purged_snaps scrub starts 2023-06-22T17:23:20.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:20 smithi039 ceph-mon[84805]: purged_snaps scrub ok 2023-06-22T17:23:20.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:20 smithi039 ceph-mon[84805]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-06-22T17:23:20.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:20 smithi039 ceph-mon[84805]: osd.4 [v2:172.21.15.101:6808/1371317064,v1:172.21.15.101:6809/1371317064] boot 2023-06-22T17:23:20.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:20 smithi039 ceph-mon[84805]: osdmap e33: 5 total, 5 up, 5 in 2023-06-22T17:23:20.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:20 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-06-22T17:23:20.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:20 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:23:20.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:20 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:23:20.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:20 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:23:21.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:21 smithi133 ceph-mon[86268]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:23:21.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:21 smithi133 ceph-mon[86268]: osdmap e34: 5 total, 5 up, 5 in 2023-06-22T17:23:21.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:21 smithi101 ceph-mon[85507]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:23:21.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:21 smithi101 ceph-mon[85507]: osdmap e34: 5 total, 5 up, 5 in 2023-06-22T17:23:21.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:21 smithi039 ceph-mon[84805]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:23:21.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:21 smithi039 ceph-mon[84805]: osdmap e34: 5 total, 5 up, 5 in 2023-06-22T17:23:22.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:22 smithi101 ceph-mon[85507]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-06-22T17:23:22.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:22 smithi133 ceph-mon[86268]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-06-22T17:23:22.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:22 smithi039 ceph-mon[84805]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-06-22T17:23:23.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:23 smithi101 ceph-mon[85507]: from='client.? 172.21.15.133:0/2282839' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6daaeb92-c73d-4c30-bb1d-18d5305a63a6"}]: dispatch 2023-06-22T17:23:23.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:23 smithi101 ceph-mon[85507]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6daaeb92-c73d-4c30-bb1d-18d5305a63a6"}]: dispatch 2023-06-22T17:23:23.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:23 smithi101 ceph-mon[85507]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6daaeb92-c73d-4c30-bb1d-18d5305a63a6"}]': finished 2023-06-22T17:23:23.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:23 smithi101 ceph-mon[85507]: osdmap e35: 6 total, 5 up, 6 in 2023-06-22T17:23:23.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:23 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-06-22T17:23:23.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:23 smithi101 ceph-mon[85507]: from='client.? 172.21.15.133:0/375769445' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:23:23.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:23 smithi133 ceph-mon[86268]: from='client.? 172.21.15.133:0/2282839' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6daaeb92-c73d-4c30-bb1d-18d5305a63a6"}]: dispatch 2023-06-22T17:23:23.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:23 smithi133 ceph-mon[86268]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6daaeb92-c73d-4c30-bb1d-18d5305a63a6"}]: dispatch 2023-06-22T17:23:23.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:23 smithi133 ceph-mon[86268]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6daaeb92-c73d-4c30-bb1d-18d5305a63a6"}]': finished 2023-06-22T17:23:23.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:23 smithi133 ceph-mon[86268]: osdmap e35: 6 total, 5 up, 6 in 2023-06-22T17:23:23.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:23 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-06-22T17:23:23.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:23 smithi133 ceph-mon[86268]: from='client.? 172.21.15.133:0/375769445' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:23:23.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:23 smithi039 ceph-mon[84805]: from='client.? 172.21.15.133:0/2282839' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6daaeb92-c73d-4c30-bb1d-18d5305a63a6"}]: dispatch 2023-06-22T17:23:23.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:23 smithi039 ceph-mon[84805]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6daaeb92-c73d-4c30-bb1d-18d5305a63a6"}]: dispatch 2023-06-22T17:23:23.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:23 smithi039 ceph-mon[84805]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6daaeb92-c73d-4c30-bb1d-18d5305a63a6"}]': finished 2023-06-22T17:23:23.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:23 smithi039 ceph-mon[84805]: osdmap e35: 6 total, 5 up, 6 in 2023-06-22T17:23:23.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:23 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-06-22T17:23:23.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:23 smithi039 ceph-mon[84805]: from='client.? 172.21.15.133:0/375769445' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:23:24.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:24 smithi133 ceph-mon[86268]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-06-22T17:23:24.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:24 smithi039 ceph-mon[84805]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-06-22T17:23:24.875 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:24 smithi101 ceph-mon[85507]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-06-22T17:23:26.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:26 smithi133 ceph-mon[86268]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-06-22T17:23:26.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:26 smithi039 ceph-mon[84805]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-06-22T17:23:26.875 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:26 smithi101 ceph-mon[85507]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-06-22T17:23:28.649 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:28 smithi133 ceph-mon[86268]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-06-22T17:23:28.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:28 smithi039 ceph-mon[84805]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-06-22T17:23:28.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:28 smithi101 ceph-mon[85507]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-06-22T17:23:29.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:29 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-06-22T17:23:29.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:29 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:23:29.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:29 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-06-22T17:23:29.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:29 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:23:29.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:29 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-06-22T17:23:29.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:29 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:23:30.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:30 smithi133 ceph-mon[86268]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-06-22T17:23:30.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:30 smithi133 ceph-mon[86268]: Deploying daemon osd.5 on smithi133 2023-06-22T17:23:30.875 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:30 smithi101 ceph-mon[85507]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-06-22T17:23:30.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:30 smithi101 ceph-mon[85507]: Deploying daemon osd.5 on smithi133 2023-06-22T17:23:30.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:30 smithi039 ceph-mon[84805]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-06-22T17:23:30.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:30 smithi039 ceph-mon[84805]: Deploying daemon osd.5 on smithi133 2023-06-22T17:23:32.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:32 smithi101 ceph-mon[85507]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-06-22T17:23:32.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:32 smithi133 ceph-mon[86268]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-06-22T17:23:32.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:32 smithi039 ceph-mon[84805]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-06-22T17:23:34.611 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:34 smithi133 ceph-mon[86268]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-06-22T17:23:34.611 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:34 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:34.611 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:34 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:34.611 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:34 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:23:34.625 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:34 smithi101 ceph-mon[85507]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-06-22T17:23:34.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:34 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:34.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:34 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:34.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:34 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:23:34.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:34 smithi039 ceph-mon[84805]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-06-22T17:23:34.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:34 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:34.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:34 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:34.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:34 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:23:36.430 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:36.428+0000 7ffaccff9700 1 -- 172.21.15.133:0/3351525205 <== mgr.14150 v2:172.21.15.39:6800/886078282 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ffad002b1b0 con 0x7ffab809b680 2023-06-22T17:23:36.430 INFO:teuthology.orchestra.run.smithi133.stdout:Created osd(s) 5 on host 'smithi133' 2023-06-22T17:23:36.432 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:36.430+0000 7ffad629e700 1 -- 172.21.15.133:0/3351525205 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7ffab809b680 msgr2=0x7ffab809db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:23:36.432 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:36.430+0000 7ffad629e700 1 --2- 172.21.15.133:0/3351525205 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7ffab809b680 0x7ffab809db30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ffad0065a30 tx=0x7ffac000b040 comp rx=0 tx=0).stop 2023-06-22T17:23:36.432 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:36.430+0000 7ffad629e700 1 -- 172.21.15.133:0/3351525205 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffad00fd140 msgr2=0x7ffad0116e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:23:36.432 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:36.430+0000 7ffad629e700 1 --2- 172.21.15.133:0/3351525205 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffad00fd140 0x7ffad0116e80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ffac8009720 tx=0x7ffac8004d10 comp rx=0 tx=0).stop 2023-06-22T17:23:36.432 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:36.430+0000 7ffad629e700 1 -- 172.21.15.133:0/3351525205 shutdown_connections 2023-06-22T17:23:36.433 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:36.430+0000 7ffad629e700 1 --2- 172.21.15.133:0/3351525205 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7ffab809b680 0x7ffab809db30 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:23:36.433 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:36.430+0000 7ffad629e700 1 --2- 172.21.15.133:0/3351525205 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffad00fd140 0x7ffad0116e80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:23:36.433 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:36.430+0000 7ffad629e700 1 --2- 172.21.15.133:0/3351525205 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffad00fbdb0 0x7ffad010f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:23:36.433 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:36.430+0000 7ffad629e700 1 --2- 172.21.15.133:0/3351525205 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ffad00fc760 0x7ffad010fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:23:36.433 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:36.430+0000 7ffad629e700 1 -- 172.21.15.133:0/3351525205 >> 172.21.15.133:0/3351525205 conn(0x7ffad00f6a50 msgr2=0x7ffad00fb570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:23:36.433 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:36.430+0000 7ffad629e700 1 -- 172.21.15.133:0/3351525205 shutdown_connections 2023-06-22T17:23:36.433 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:23:36.430+0000 7ffad629e700 1 -- 172.21.15.133:0/3351525205 wait complete. 2023-06-22T17:23:36.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:36 smithi101 ceph-mon[85507]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-06-22T17:23:36.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:36 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:36.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:36 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:36.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:36 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:23:36.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:36 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:23:36.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:36 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:36.657 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:36 smithi133 ceph-mon[86268]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-06-22T17:23:36.657 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:36 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:36.658 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:36 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:36.658 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:36 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:23:36.658 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:36 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:23:36.658 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:36 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:36.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:36 smithi039 ceph-mon[84805]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-06-22T17:23:36.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:36 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:36.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:36 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:36.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:36 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:23:36.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:36 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:23:36.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:36 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:36.913 DEBUG:teuthology.orchestra.run.smithi133:osd.5> sudo journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.5.service 2023-06-22T17:23:36.915 INFO:tasks.cephadm:Deploying osd.6 on smithi039 with /dev/vg_nvme/lv_2... 2023-06-22T17:23:36.915 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-06-22T17:23:36.942 INFO:journalctl@ceph.osd.5.smithi133.stdout:-- Logs begin at Thu 2023-06-22 17:06:34 UTC. -- 2023-06-22T17:23:37.526 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-22T17:23:37.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:37 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:37.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:37 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:37.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:37 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:23:37.671 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:23:37 smithi133 conmon[100972]: 2023-06-22T17:23:37.532+0000 7efe8ead1540 -1 osd.5 0 log_to_monitors true 2023-06-22T17:23:37.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:37 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:37.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:37 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:37.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:37 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:23:37.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:37 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:37.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:37 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:37.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:37 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:23:38.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:38 smithi039 ceph-mon[84805]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-06-22T17:23:38.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:38 smithi039 ceph-mon[84805]: from='osd.5 [v2:172.21.15.133:6808/747181981,v1:172.21.15.133:6809/747181981]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-06-22T17:23:38.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:38 smithi039 ceph-mon[84805]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-06-22T17:23:38.695 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:38 smithi133 ceph-mon[86268]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-06-22T17:23:38.696 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:38 smithi133 ceph-mon[86268]: from='osd.5 [v2:172.21.15.133:6808/747181981,v1:172.21.15.133:6809/747181981]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-06-22T17:23:38.696 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:38 smithi133 ceph-mon[86268]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-06-22T17:23:38.875 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:38 smithi101 ceph-mon[85507]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-06-22T17:23:38.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:38 smithi101 ceph-mon[85507]: from='osd.5 [v2:172.21.15.133:6808/747181981,v1:172.21.15.133:6809/747181981]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-06-22T17:23:38.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:38 smithi101 ceph-mon[85507]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-06-22T17:23:39.025 INFO:teuthology.orchestra.run.smithi039.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-06-22T17:23:39.025 INFO:teuthology.orchestra.run.smithi039.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-06-22T17:23:39.026 INFO:teuthology.orchestra.run.smithi039.stderr: stderr: 10+0 records in 2023-06-22T17:23:39.026 INFO:teuthology.orchestra.run.smithi039.stderr:10+0 records out 2023-06-22T17:23:39.026 INFO:teuthology.orchestra.run.smithi039.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0100395 s, 1.0 GB/s 2023-06-22T17:23:39.026 INFO:teuthology.orchestra.run.smithi039.stderr:--> Zapping successful for: 2023-06-22T17:23:39.473 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph orch daemon add osd smithi039:vg_nvme/lv_2 2023-06-22T17:23:39.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:39 smithi101 ceph-mon[85507]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-06-22T17:23:39.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:39 smithi101 ceph-mon[85507]: osdmap e36: 6 total, 5 up, 6 in 2023-06-22T17:23:39.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:39 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-06-22T17:23:39.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:39 smithi101 ceph-mon[85507]: from='osd.5 [v2:172.21.15.133:6808/747181981,v1:172.21.15.133:6809/747181981]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2023-06-22T17:23:39.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:39 smithi101 ceph-mon[85507]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2023-06-22T17:23:39.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:39 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:39.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:39 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:39.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:39 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:23:39.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:39 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:23:39.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:39 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:39.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:39 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:23:39.878 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:39 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:23:39.878 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:39 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:39.920 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:23:39 smithi133 conmon[100972]: 2023-06-22T17:23:39.430+0000 7efe852b6700 -1 osd.5 0 waiting for initial osdmap 2023-06-22T17:23:39.920 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:23:39 smithi133 conmon[100972]: 2023-06-22T17:23:39.439+0000 7efe7f0af700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-22T17:23:39.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:39 smithi133 ceph-mon[86268]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-06-22T17:23:39.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:39 smithi133 ceph-mon[86268]: osdmap e36: 6 total, 5 up, 6 in 2023-06-22T17:23:39.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:39 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-06-22T17:23:39.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:39 smithi133 ceph-mon[86268]: from='osd.5 [v2:172.21.15.133:6808/747181981,v1:172.21.15.133:6809/747181981]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2023-06-22T17:23:39.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:39 smithi133 ceph-mon[86268]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2023-06-22T17:23:39.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:39 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:39.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:39 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:39.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:39 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:23:39.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:39 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:23:39.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:39 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:39.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:39 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:23:39.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:39 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:23:39.923 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:39 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:39.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:39 smithi039 ceph-mon[84805]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-06-22T17:23:39.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:39 smithi039 ceph-mon[84805]: osdmap e36: 6 total, 5 up, 6 in 2023-06-22T17:23:39.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:39 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-06-22T17:23:39.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:39 smithi039 ceph-mon[84805]: from='osd.5 [v2:172.21.15.133:6808/747181981,v1:172.21.15.133:6809/747181981]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2023-06-22T17:23:39.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:39 smithi039 ceph-mon[84805]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2023-06-22T17:23:39.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:39 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:39.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:39 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:39.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:39 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:23:39.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:39 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:23:39.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:39 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:39.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:39 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:23:39.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:39 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:23:39.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:39 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:40.067 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-22T17:23:40.621 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:40 smithi039 ceph-mon[84805]: Detected new or changed devices on smithi133 2023-06-22T17:23:40.621 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:40 smithi039 ceph-mon[84805]: Adjusting osd_memory_target on smithi133 to 8584M 2023-06-22T17:23:40.621 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:40 smithi039 ceph-mon[84805]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-06-22T17:23:40.622 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:40 smithi039 ceph-mon[84805]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]': finished 2023-06-22T17:23:40.622 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:40 smithi039 ceph-mon[84805]: osdmap e37: 6 total, 5 up, 6 in 2023-06-22T17:23:40.622 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:40 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-06-22T17:23:40.622 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:40 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-06-22T17:23:40.622 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:40 smithi039 ceph-mon[84805]: osd.5 [v2:172.21.15.133:6808/747181981,v1:172.21.15.133:6809/747181981] boot 2023-06-22T17:23:40.622 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:40 smithi039 ceph-mon[84805]: osdmap e38: 6 total, 6 up, 6 in 2023-06-22T17:23:40.622 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:40 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-06-22T17:23:40.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:40 smithi101 ceph-mon[85507]: Detected new or changed devices on smithi133 2023-06-22T17:23:40.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:40 smithi101 ceph-mon[85507]: Adjusting osd_memory_target on smithi133 to 8584M 2023-06-22T17:23:40.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:40 smithi101 ceph-mon[85507]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-06-22T17:23:40.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:40 smithi101 ceph-mon[85507]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]': finished 2023-06-22T17:23:40.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:40 smithi101 ceph-mon[85507]: osdmap e37: 6 total, 5 up, 6 in 2023-06-22T17:23:40.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:40 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-06-22T17:23:40.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:40 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-06-22T17:23:40.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:40 smithi101 ceph-mon[85507]: osd.5 [v2:172.21.15.133:6808/747181981,v1:172.21.15.133:6809/747181981] boot 2023-06-22T17:23:40.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:40 smithi101 ceph-mon[85507]: osdmap e38: 6 total, 6 up, 6 in 2023-06-22T17:23:40.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:40 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-06-22T17:23:40.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:40 smithi133 ceph-mon[86268]: Detected new or changed devices on smithi133 2023-06-22T17:23:40.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:40 smithi133 ceph-mon[86268]: Adjusting osd_memory_target on smithi133 to 8584M 2023-06-22T17:23:40.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:40 smithi133 ceph-mon[86268]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-06-22T17:23:40.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:40 smithi133 ceph-mon[86268]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]': finished 2023-06-22T17:23:40.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:40 smithi133 ceph-mon[86268]: osdmap e37: 6 total, 5 up, 6 in 2023-06-22T17:23:40.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:40 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-06-22T17:23:40.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:40 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-06-22T17:23:40.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:40 smithi133 ceph-mon[86268]: osd.5 [v2:172.21.15.133:6808/747181981,v1:172.21.15.133:6809/747181981] boot 2023-06-22T17:23:40.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:40 smithi133 ceph-mon[86268]: osdmap e38: 6 total, 6 up, 6 in 2023-06-22T17:23:40.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:40 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-06-22T17:23:41.328 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.284+0000 7efd5debf700 1 -- 172.21.15.39:0/1652972911 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd580fb240 msgr2=0x7efd581023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:23:41.328 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.284+0000 7efd5debf700 1 --2- 172.21.15.39:0/1652972911 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd580fb240 0x7efd581023a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7efd480049c0 tx=0x7efd48009e30 comp rx=0 tx=0).stop 2023-06-22T17:23:41.328 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.284+0000 7efd5debf700 1 -- 172.21.15.39:0/1652972911 shutdown_connections 2023-06-22T17:23:41.329 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.284+0000 7efd5debf700 1 --2- 172.21.15.39:0/1652972911 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efd581028e0 0x7efd58104d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:23:41.329 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.284+0000 7efd5debf700 1 --2- 172.21.15.39:0/1652972911 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efd580fa860 0x7efd580fac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:23:41.329 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.284+0000 7efd5debf700 1 --2- 172.21.15.39:0/1652972911 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd580fb240 0x7efd581023a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:23:41.329 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.284+0000 7efd5debf700 1 -- 172.21.15.39:0/1652972911 >> 172.21.15.39:0/1652972911 conn(0x7efd580f6580 msgr2=0x7efd580f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:23:41.329 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.284+0000 7efd5debf700 1 -- 172.21.15.39:0/1652972911 shutdown_connections 2023-06-22T17:23:41.329 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.284+0000 7efd5debf700 1 -- 172.21.15.39:0/1652972911 wait complete. 2023-06-22T17:23:41.329 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.285+0000 7efd5debf700 1 Processor -- start 2023-06-22T17:23:41.330 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.285+0000 7efd5debf700 1 -- start start 2023-06-22T17:23:41.330 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.285+0000 7efd5debf700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd580fa860 0x7efd5810d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:23:41.330 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.285+0000 7efd5debf700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efd580fb240 0x7efd5810de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:23:41.330 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.285+0000 7efd5debf700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efd581028e0 0x7efd58114eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:23:41.330 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.285+0000 7efd5debf700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd581154c0 con 0x7efd580fa860 2023-06-22T17:23:41.330 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.285+0000 7efd5debf700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd58115630 con 0x7efd581028e0 2023-06-22T17:23:41.330 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.285+0000 7efd5debf700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd581157f0 con 0x7efd580fb240 2023-06-22T17:23:41.330 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.285+0000 7efd577fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd580fa860 0x7efd5810d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:23:41.331 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.285+0000 7efd577fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd580fa860 0x7efd5810d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52060/0 (socket says 172.21.15.39:52060) 2023-06-22T17:23:41.331 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.285+0000 7efd577fe700 1 -- 172.21.15.39:0/2832773400 learned_addr learned my addr 172.21.15.39:0/2832773400 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:23:41.331 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.285+0000 7efd56ffd700 1 --2- 172.21.15.39:0/2832773400 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efd580fb240 0x7efd5810de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:23:41.331 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.286+0000 7efd57fff700 1 --2- 172.21.15.39:0/2832773400 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efd581028e0 0x7efd58114eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:23:41.331 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.286+0000 7efd577fe700 1 -- 172.21.15.39:0/2832773400 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efd580fb240 msgr2=0x7efd5810de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:23:41.332 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.286+0000 7efd577fe700 1 --2- 172.21.15.39:0/2832773400 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efd580fb240 0x7efd5810de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:23:41.332 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.286+0000 7efd577fe700 1 -- 172.21.15.39:0/2832773400 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efd581028e0 msgr2=0x7efd58114eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:23:41.332 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.286+0000 7efd577fe700 1 --2- 172.21.15.39:0/2832773400 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efd581028e0 0x7efd58114eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:23:41.332 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.286+0000 7efd577fe700 1 -- 172.21.15.39:0/2832773400 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd48005040 con 0x7efd580fa860 2023-06-22T17:23:41.332 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.286+0000 7efd577fe700 1 --2- 172.21.15.39:0/2832773400 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd580fa860 0x7efd5810d8e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7efd40002b00 tx=0x7efd400046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:23:41.332 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.286+0000 7efd4ffff700 1 -- 172.21.15.39:0/2832773400 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd40007cc0 con 0x7efd580fa860 2023-06-22T17:23:41.332 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.286+0000 7efd5debf700 1 -- 172.21.15.39:0/2832773400 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd58115ad0 con 0x7efd580fa860 2023-06-22T17:23:41.332 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.286+0000 7efd4ffff700 1 -- 172.21.15.39:0/2832773400 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7efd4000f040 con 0x7efd580fa860 2023-06-22T17:23:41.333 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.286+0000 7efd4ffff700 1 -- 172.21.15.39:0/2832773400 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd4000da90 con 0x7efd580fa860 2023-06-22T17:23:41.333 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.286+0000 7efd5debf700 1 -- 172.21.15.39:0/2832773400 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd58115fc0 con 0x7efd580fa860 2023-06-22T17:23:41.333 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.287+0000 7efd5debf700 1 -- 172.21.15.39:0/2832773400 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd581085a0 con 0x7efd580fa860 2023-06-22T17:23:41.333 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.288+0000 7efd4ffff700 1 -- 172.21.15.39:0/2832773400 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7efd400090d0 con 0x7efd580fa860 2023-06-22T17:23:41.333 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.289+0000 7efd4ffff700 1 --2- 172.21.15.39:0/2832773400 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7efd4409b680 0x7efd4409db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:23:41.333 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.289+0000 7efd4ffff700 1 -- 172.21.15.39:0/2832773400 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4562+0+0 (secure 0 0 0) 0x7efd400c43f0 con 0x7efd580fa860 2023-06-22T17:23:41.333 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.289+0000 7efd56ffd700 1 --2- 172.21.15.39:0/2832773400 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7efd4409b680 0x7efd4409db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:23:41.333 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.289+0000 7efd56ffd700 1 --2- 172.21.15.39:0/2832773400 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7efd4409b680 0x7efd4409db30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7efd4800b3b0 tx=0x7efd48009e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:23:41.334 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.292+0000 7efd4ffff700 1 -- 172.21.15.39:0/2832773400 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7efd40090af0 con 0x7efd580fa860 2023-06-22T17:23:41.500 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:41.499+0000 7efd5debf700 1 -- 172.21.15.39:0/2832773400 --> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7efd581161b0 con 0x7efd4409b680 2023-06-22T17:23:41.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:41 smithi101 ceph-mon[85507]: purged_snaps scrub starts 2023-06-22T17:23:41.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:41 smithi101 ceph-mon[85507]: purged_snaps scrub ok 2023-06-22T17:23:41.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:41 smithi101 ceph-mon[85507]: osdmap e39: 6 total, 6 up, 6 in 2023-06-22T17:23:41.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:41 smithi133 ceph-mon[86268]: purged_snaps scrub starts 2023-06-22T17:23:41.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:41 smithi133 ceph-mon[86268]: purged_snaps scrub ok 2023-06-22T17:23:41.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:41 smithi133 ceph-mon[86268]: osdmap e39: 6 total, 6 up, 6 in 2023-06-22T17:23:41.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:41 smithi039 ceph-mon[84805]: purged_snaps scrub starts 2023-06-22T17:23:41.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:41 smithi039 ceph-mon[84805]: purged_snaps scrub ok 2023-06-22T17:23:41.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:41 smithi039 ceph-mon[84805]: osdmap e39: 6 total, 6 up, 6 in 2023-06-22T17:23:42.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:42 smithi039 ceph-mon[84805]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-06-22T17:23:42.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:42 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:23:42.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:42 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:23:42.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:42 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:23:42.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:42 smithi039 ceph-mon[84805]: osdmap e40: 6 total, 6 up, 6 in 2023-06-22T17:23:42.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:42 smithi101 ceph-mon[85507]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-06-22T17:23:42.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:42 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:23:42.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:42 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:23:42.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:42 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:23:42.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:42 smithi101 ceph-mon[85507]: osdmap e40: 6 total, 6 up, 6 in 2023-06-22T17:23:42.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:42 smithi133 ceph-mon[86268]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-06-22T17:23:42.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:42 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:23:42.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:42 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:23:42.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:42 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:23:42.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:42 smithi133 ceph-mon[86268]: osdmap e40: 6 total, 6 up, 6 in 2023-06-22T17:23:43.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:43 smithi039 ceph-mon[84805]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:23:43.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:43 smithi101 ceph-mon[85507]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:23:43.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:43 smithi133 ceph-mon[86268]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:23:44.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:44 smithi101 ceph-mon[85507]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-06-22T17:23:44.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:44 smithi101 ceph-mon[85507]: osdmap e41: 6 total, 6 up, 6 in 2023-06-22T17:23:44.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:44 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/2634363718' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "21f46569-8123-4bdd-b3c5-4cc8bc271899"}]: dispatch 2023-06-22T17:23:44.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:44 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/2634363718' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "21f46569-8123-4bdd-b3c5-4cc8bc271899"}]': finished 2023-06-22T17:23:44.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:44 smithi101 ceph-mon[85507]: osdmap e42: 7 total, 6 up, 7 in 2023-06-22T17:23:44.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:44 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-22T17:23:44.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:44 smithi133 ceph-mon[86268]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-06-22T17:23:44.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:44 smithi133 ceph-mon[86268]: osdmap e41: 6 total, 6 up, 6 in 2023-06-22T17:23:44.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:44 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/2634363718' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "21f46569-8123-4bdd-b3c5-4cc8bc271899"}]: dispatch 2023-06-22T17:23:44.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:44 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/2634363718' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "21f46569-8123-4bdd-b3c5-4cc8bc271899"}]': finished 2023-06-22T17:23:44.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:44 smithi133 ceph-mon[86268]: osdmap e42: 7 total, 6 up, 7 in 2023-06-22T17:23:44.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:44 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-22T17:23:44.932 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:44 smithi039 ceph-mon[84805]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-06-22T17:23:44.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:44 smithi039 ceph-mon[84805]: osdmap e41: 6 total, 6 up, 6 in 2023-06-22T17:23:44.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:44 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/2634363718' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "21f46569-8123-4bdd-b3c5-4cc8bc271899"}]: dispatch 2023-06-22T17:23:44.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:44 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/2634363718' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "21f46569-8123-4bdd-b3c5-4cc8bc271899"}]': finished 2023-06-22T17:23:44.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:44 smithi039 ceph-mon[84805]: osdmap e42: 7 total, 6 up, 7 in 2023-06-22T17:23:44.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:44 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-22T17:23:46.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:45 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/865212946' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:23:46.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:45 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/865212946' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:23:46.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:45 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/865212946' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:23:47.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:46 smithi101 ceph-mon[85507]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-06-22T17:23:47.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:46 smithi133 ceph-mon[86268]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-06-22T17:23:47.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:46 smithi039 ceph-mon[84805]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-06-22T17:23:48.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:47 smithi101 ceph-mon[85507]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-06-22T17:23:48.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:47 smithi133 ceph-mon[86268]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-06-22T17:23:48.182 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:47 smithi039 ceph-mon[84805]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-06-22T17:23:50.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:50 smithi039 ceph-mon[84805]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 83 KiB/s, 0 objects/s recovering 2023-06-22T17:23:50.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:50 smithi133 ceph-mon[86268]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 83 KiB/s, 0 objects/s recovering 2023-06-22T17:23:50.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:50 smithi101 ceph-mon[85507]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 83 KiB/s, 0 objects/s recovering 2023-06-22T17:23:52.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:52 smithi133 ceph-mon[86268]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-06-22T17:23:52.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:52 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-06-22T17:23:52.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:52 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:23:52.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:52 smithi039 ceph-mon[84805]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-06-22T17:23:52.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:52 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-06-22T17:23:52.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:52 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:23:52.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:52 smithi101 ceph-mon[85507]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-06-22T17:23:52.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:52 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-06-22T17:23:52.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:52 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:23:53.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:53 smithi133 ceph-mon[86268]: Deploying daemon osd.6 on smithi039 2023-06-22T17:23:53.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:53 smithi039 ceph-mon[84805]: Deploying daemon osd.6 on smithi039 2023-06-22T17:23:53.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:53 smithi101 ceph-mon[85507]: Deploying daemon osd.6 on smithi039 2023-06-22T17:23:54.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:54 smithi039 ceph-mon[84805]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-06-22T17:23:54.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:54 smithi133 ceph-mon[86268]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-06-22T17:23:54.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:54 smithi101 ceph-mon[85507]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-06-22T17:23:56.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:56 smithi133 ceph-mon[86268]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 51 KiB/s, 0 objects/s recovering 2023-06-22T17:23:56.682 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:56 smithi039 ceph-mon[84805]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 51 KiB/s, 0 objects/s recovering 2023-06-22T17:23:56.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:56 smithi101 ceph-mon[85507]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 51 KiB/s, 0 objects/s recovering 2023-06-22T17:23:57.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:57 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:23:57.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:57 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:57.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:57 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:57.673 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:57 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:23:57.673 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:57 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:57.673 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:57 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:57.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:57 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:23:57.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:57 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:57.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:57 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:23:58.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:58 smithi133 ceph-mon[86268]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-06-22T17:23:58.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:58 smithi039 ceph-mon[84805]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-06-22T17:23:58.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:58 smithi101 ceph-mon[85507]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-06-22T17:23:59.554 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:59.553+0000 7efd4ffff700 1 -- 172.21.15.39:0/2832773400 <== mgr.14150 v2:172.21.15.39:6800/886078282 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7efd581161b0 con 0x7efd4409b680 2023-06-22T17:23:59.555 INFO:teuthology.orchestra.run.smithi039.stdout:Created osd(s) 6 on host 'smithi039' 2023-06-22T17:23:59.556 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:59.555+0000 7efd5debf700 1 -- 172.21.15.39:0/2832773400 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7efd4409b680 msgr2=0x7efd4409db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:23:59.556 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:59.555+0000 7efd5debf700 1 --2- 172.21.15.39:0/2832773400 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7efd4409b680 0x7efd4409db30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7efd4800b3b0 tx=0x7efd48009e30 comp rx=0 tx=0).stop 2023-06-22T17:23:59.557 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:59.555+0000 7efd5debf700 1 -- 172.21.15.39:0/2832773400 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd580fa860 msgr2=0x7efd5810d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:23:59.557 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:59.555+0000 7efd5debf700 1 --2- 172.21.15.39:0/2832773400 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd580fa860 0x7efd5810d8e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7efd40002b00 tx=0x7efd400046b0 comp rx=0 tx=0).stop 2023-06-22T17:23:59.557 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:59.555+0000 7efd5debf700 1 -- 172.21.15.39:0/2832773400 shutdown_connections 2023-06-22T17:23:59.557 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:59.555+0000 7efd5debf700 1 --2- 172.21.15.39:0/2832773400 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7efd4409b680 0x7efd4409db30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:23:59.557 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:59.555+0000 7efd5debf700 1 --2- 172.21.15.39:0/2832773400 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efd581028e0 0x7efd58114eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:23:59.557 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:59.555+0000 7efd5debf700 1 --2- 172.21.15.39:0/2832773400 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd580fa860 0x7efd5810d8e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:23:59.557 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:59.555+0000 7efd5debf700 1 --2- 172.21.15.39:0/2832773400 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efd580fb240 0x7efd5810de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:23:59.558 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:59.555+0000 7efd5debf700 1 -- 172.21.15.39:0/2832773400 >> 172.21.15.39:0/2832773400 conn(0x7efd580f6580 msgr2=0x7efd580f71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:23:59.558 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:59.555+0000 7efd5debf700 1 -- 172.21.15.39:0/2832773400 shutdown_connections 2023-06-22T17:23:59.558 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:23:59.555+0000 7efd5debf700 1 -- 172.21.15.39:0/2832773400 wait complete. 2023-06-22T17:24:00.033 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:59 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:00.033 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:59 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:00.033 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:59 smithi039 ceph-mon[84805]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-06-22T17:24:00.034 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:59 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:00.034 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:59 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:24:00.034 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:59 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:00.034 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:59 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:00.034 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:59 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:00.034 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:23:59 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:24:00.035 DEBUG:teuthology.orchestra.run.smithi039:osd.6> sudo journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.6.service 2023-06-22T17:24:00.037 INFO:tasks.cephadm:Deploying osd.7 on smithi101 with /dev/vg_nvme/lv_2... 2023-06-22T17:24:00.038 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-06-22T17:24:00.080 INFO:journalctl@ceph.osd.6.smithi039.stdout:-- Logs begin at Thu 2023-06-22 17:07:05 UTC. -- 2023-06-22T17:24:00.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:59 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:00.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:59 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:00.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:59 smithi133 ceph-mon[86268]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-06-22T17:24:00.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:59 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:00.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:59 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:24:00.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:59 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:00.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:59 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:00.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:59 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:00.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:23:59 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:24:00.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:59 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:00.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:59 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:00.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:59 smithi101 ceph-mon[85507]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-06-22T17:24:00.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:59 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:00.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:59 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:24:00.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:59 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:00.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:59 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:00.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:59 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:00.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:23:59 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:24:00.633 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.b/config 2023-06-22T17:24:00.905 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:24:00 smithi039 conmon[115622]: 2023-06-22T17:24:00.483+0000 7fb4cc2b1540 -1 osd.6 0 log_to_monitors true 2023-06-22T17:24:01.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:00 smithi101 ceph-mon[85507]: from='osd.6 [v2:172.21.15.39:6818/1221216882,v1:172.21.15.39:6819/1221216882]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-06-22T17:24:01.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:00 smithi133 ceph-mon[86268]: from='osd.6 [v2:172.21.15.39:6818/1221216882,v1:172.21.15.39:6819/1221216882]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-06-22T17:24:01.173 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:00 smithi039 ceph-mon[84805]: from='osd.6 [v2:172.21.15.39:6818/1221216882,v1:172.21.15.39:6819/1221216882]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-06-22T17:24:02.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:01 smithi101 ceph-mon[85507]: from='osd.6 [v2:172.21.15.39:6818/1221216882,v1:172.21.15.39:6819/1221216882]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-06-22T17:24:02.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:01 smithi101 ceph-mon[85507]: osdmap e43: 7 total, 6 up, 7 in 2023-06-22T17:24:02.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:01 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-22T17:24:02.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:01 smithi101 ceph-mon[85507]: from='osd.6 [v2:172.21.15.39:6818/1221216882,v1:172.21.15.39:6819/1221216882]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-06-22T17:24:02.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:01 smithi101 ceph-mon[85507]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-06-22T17:24:02.183 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:24:01 smithi039 conmon[115622]: 2023-06-22T17:24:01.928+0000 7fb4c2a96700 -1 osd.6 0 waiting for initial osdmap 2023-06-22T17:24:02.183 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:24:01 smithi039 conmon[115622]: 2023-06-22T17:24:01.941+0000 7fb4bc88f700 -1 osd.6 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-22T17:24:02.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:01 smithi039 ceph-mon[84805]: from='osd.6 [v2:172.21.15.39:6818/1221216882,v1:172.21.15.39:6819/1221216882]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-06-22T17:24:02.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:01 smithi039 ceph-mon[84805]: osdmap e43: 7 total, 6 up, 7 in 2023-06-22T17:24:02.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:01 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-22T17:24:02.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:01 smithi039 ceph-mon[84805]: from='osd.6 [v2:172.21.15.39:6818/1221216882,v1:172.21.15.39:6819/1221216882]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-06-22T17:24:02.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:01 smithi039 ceph-mon[84805]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-06-22T17:24:02.221 INFO:teuthology.orchestra.run.smithi101.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-06-22T17:24:02.221 INFO:teuthology.orchestra.run.smithi101.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-06-22T17:24:02.221 INFO:teuthology.orchestra.run.smithi101.stderr: stderr: 10+0 records in 2023-06-22T17:24:02.221 INFO:teuthology.orchestra.run.smithi101.stderr:10+0 records out 2023-06-22T17:24:02.222 INFO:teuthology.orchestra.run.smithi101.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130415 s, 804 MB/s 2023-06-22T17:24:02.222 INFO:teuthology.orchestra.run.smithi101.stderr:--> Zapping successful for: 2023-06-22T17:24:02.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:01 smithi133 ceph-mon[86268]: from='osd.6 [v2:172.21.15.39:6818/1221216882,v1:172.21.15.39:6819/1221216882]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-06-22T17:24:02.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:01 smithi133 ceph-mon[86268]: osdmap e43: 7 total, 6 up, 7 in 2023-06-22T17:24:02.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:01 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-22T17:24:02.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:01 smithi133 ceph-mon[86268]: from='osd.6 [v2:172.21.15.39:6818/1221216882,v1:172.21.15.39:6819/1221216882]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-06-22T17:24:02.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:01 smithi133 ceph-mon[86268]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-06-22T17:24:02.674 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph orch daemon add osd smithi101:vg_nvme/lv_2 2023-06-22T17:24:03.255 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.b/config 2023-06-22T17:24:03.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:02 smithi101 ceph-mon[85507]: from='osd.6 [v2:172.21.15.39:6818/1221216882,v1:172.21.15.39:6819/1221216882]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2023-06-22T17:24:03.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:02 smithi101 ceph-mon[85507]: osdmap e44: 7 total, 6 up, 7 in 2023-06-22T17:24:03.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:02 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-22T17:24:03.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:02 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-22T17:24:03.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:02 smithi101 ceph-mon[85507]: Detected new or changed devices on smithi039 2023-06-22T17:24:03.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:02 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:03.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:02 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:03.378 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:02 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:24:03.378 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:02 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:24:03.378 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:02 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:24:03.378 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:02 smithi101 ceph-mon[85507]: Adjusting osd_memory_target on smithi039 to 5722M 2023-06-22T17:24:03.378 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:02 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:03.378 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:02 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:03.378 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:02 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:24:03.378 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:02 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:03.379 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:02 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-22T17:24:03.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:02 smithi133 ceph-mon[86268]: from='osd.6 [v2:172.21.15.39:6818/1221216882,v1:172.21.15.39:6819/1221216882]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2023-06-22T17:24:03.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:02 smithi133 ceph-mon[86268]: osdmap e44: 7 total, 6 up, 7 in 2023-06-22T17:24:03.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-22T17:24:03.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-22T17:24:03.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:02 smithi133 ceph-mon[86268]: Detected new or changed devices on smithi039 2023-06-22T17:24:03.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:03.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:03.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:24:03.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:24:03.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:24:03.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:02 smithi133 ceph-mon[86268]: Adjusting osd_memory_target on smithi039 to 5722M 2023-06-22T17:24:03.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:03.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:03.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:24:03.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:03.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-22T17:24:03.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:02 smithi039 ceph-mon[84805]: from='osd.6 [v2:172.21.15.39:6818/1221216882,v1:172.21.15.39:6819/1221216882]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2023-06-22T17:24:03.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:02 smithi039 ceph-mon[84805]: osdmap e44: 7 total, 6 up, 7 in 2023-06-22T17:24:03.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:02 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-22T17:24:03.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:02 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-22T17:24:03.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:02 smithi039 ceph-mon[84805]: Detected new or changed devices on smithi039 2023-06-22T17:24:03.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:02 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:03.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:02 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:03.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:02 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:24:03.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:02 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:24:03.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:02 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:24:03.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:02 smithi039 ceph-mon[84805]: Adjusting osd_memory_target on smithi039 to 5722M 2023-06-22T17:24:03.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:02 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:03.435 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:02 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:03.435 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:02 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:24:03.435 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:02 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:03.435 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:02 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-22T17:24:04.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:03 smithi101 ceph-mon[85507]: purged_snaps scrub starts 2023-06-22T17:24:04.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:03 smithi101 ceph-mon[85507]: purged_snaps scrub ok 2023-06-22T17:24:04.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:03 smithi101 ceph-mon[85507]: osd.6 [v2:172.21.15.39:6818/1221216882,v1:172.21.15.39:6819/1221216882] boot 2023-06-22T17:24:04.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:03 smithi101 ceph-mon[85507]: osdmap e45: 7 total, 7 up, 7 in 2023-06-22T17:24:04.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:03 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-22T17:24:04.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:03 smithi101 ceph-mon[85507]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-06-22T17:24:04.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:03 smithi133 ceph-mon[86268]: purged_snaps scrub starts 2023-06-22T17:24:04.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:03 smithi133 ceph-mon[86268]: purged_snaps scrub ok 2023-06-22T17:24:04.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:03 smithi133 ceph-mon[86268]: osd.6 [v2:172.21.15.39:6818/1221216882,v1:172.21.15.39:6819/1221216882] boot 2023-06-22T17:24:04.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:03 smithi133 ceph-mon[86268]: osdmap e45: 7 total, 7 up, 7 in 2023-06-22T17:24:04.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:03 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-22T17:24:04.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:03 smithi133 ceph-mon[86268]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-06-22T17:24:04.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:03 smithi039 ceph-mon[84805]: purged_snaps scrub starts 2023-06-22T17:24:04.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:03 smithi039 ceph-mon[84805]: purged_snaps scrub ok 2023-06-22T17:24:04.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:03 smithi039 ceph-mon[84805]: osd.6 [v2:172.21.15.39:6818/1221216882,v1:172.21.15.39:6819/1221216882] boot 2023-06-22T17:24:04.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:03 smithi039 ceph-mon[84805]: osdmap e45: 7 total, 7 up, 7 in 2023-06-22T17:24:04.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:03 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-22T17:24:04.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:03 smithi039 ceph-mon[84805]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-06-22T17:24:04.464 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.462+0000 7f2ae5dd6700 1 -- 172.21.15.101:0/3156841460 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2ae00fbaf0 msgr2=0x7f2ae00fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:24:04.464 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.462+0000 7f2ae5dd6700 1 --2- 172.21.15.101:0/3156841460 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2ae00fbaf0 0x7f2ae00fbf00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2ad0009d60 tx=0x7f2ad0004d10 comp rx=0 tx=0).stop 2023-06-22T17:24:04.464 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.462+0000 7f2ae5dd6700 1 -- 172.21.15.101:0/3156841460 shutdown_connections 2023-06-22T17:24:04.464 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.462+0000 7f2ae5dd6700 1 --2- 172.21.15.101:0/3156841460 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ae00fd750 0x7f2ae0103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:24:04.464 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.462+0000 7f2ae5dd6700 1 --2- 172.21.15.101:0/3156841460 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2ae00fbaf0 0x7f2ae00fbf00 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:24:04.465 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.462+0000 7f2ae5dd6700 1 --2- 172.21.15.101:0/3156841460 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ae00fcda0 0x7f2ae00fd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:24:04.465 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.462+0000 7f2ae5dd6700 1 -- 172.21.15.101:0/3156841460 >> 172.21.15.101:0/3156841460 conn(0x7f2ae00f7210 msgr2=0x7f2ae00f9680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:24:04.465 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.462+0000 7f2ae5dd6700 1 -- 172.21.15.101:0/3156841460 shutdown_connections 2023-06-22T17:24:04.465 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.462+0000 7f2ae5dd6700 1 -- 172.21.15.101:0/3156841460 wait complete. 2023-06-22T17:24:04.466 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.462+0000 7f2ae5dd6700 1 Processor -- start 2023-06-22T17:24:04.467 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.463+0000 7f2ae5dd6700 1 -- start start 2023-06-22T17:24:04.467 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.463+0000 7f2ae5dd6700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2ae00fbaf0 0x7f2ae0065f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:24:04.467 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.463+0000 7f2ae5dd6700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ae00fcda0 0x7f2ae0112870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:24:04.467 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.463+0000 7f2ae5dd6700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ae00fd750 0x7f2ae0114eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:24:04.467 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.463+0000 7f2ae5dd6700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae0115650 con 0x7f2ae00fcda0 2023-06-22T17:24:04.467 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.463+0000 7f2ae5dd6700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae01157c0 con 0x7f2ae00fbaf0 2023-06-22T17:24:04.468 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.463+0000 7f2ae5dd6700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae0115980 con 0x7f2ae00fd750 2023-06-22T17:24:04.468 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.463+0000 7f2adf7fe700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2ae00fbaf0 0x7f2ae0065f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:24:04.468 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.463+0000 7f2adf7fe700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2ae00fbaf0 0x7f2ae0065f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:56942/0 (socket says 172.21.15.101:56942) 2023-06-22T17:24:04.468 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.463+0000 7f2adf7fe700 1 -- 172.21.15.101:0/1029756668 learned_addr learned my addr 172.21.15.101:0/1029756668 (peer_addr_for_me v2:172.21.15.101:0/0) 2023-06-22T17:24:04.468 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.463+0000 7f2adeffd700 1 --2- 172.21.15.101:0/1029756668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ae00fcda0 0x7f2ae0112870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:24:04.468 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.463+0000 7f2adf7fe700 1 -- 172.21.15.101:0/1029756668 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ae00fd750 msgr2=0x7f2ae0114eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:24:04.468 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.463+0000 7f2adf7fe700 1 --2- 172.21.15.101:0/1029756668 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ae00fd750 0x7f2ae0114eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:24:04.469 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.463+0000 7f2adf7fe700 1 -- 172.21.15.101:0/1029756668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ae00fcda0 msgr2=0x7f2ae0112870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:24:04.469 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.463+0000 7f2adf7fe700 1 --2- 172.21.15.101:0/1029756668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ae00fcda0 0x7f2ae0112870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:24:04.469 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.463+0000 7f2adf7fe700 1 -- 172.21.15.101:0/1029756668 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ad0009940 con 0x7f2ae00fbaf0 2023-06-22T17:24:04.469 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.463+0000 7f2adeffd700 1 --2- 172.21.15.101:0/1029756668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ae00fcda0 0x7f2ae0112870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T17:24:04.469 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.463+0000 7f2adf7fe700 1 --2- 172.21.15.101:0/1029756668 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2ae00fbaf0 0x7f2ae0065f00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2ad0007c50 tx=0x7f2ad0004d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:24:04.469 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.464+0000 7f2ac7fff700 1 -- 172.21.15.101:0/1029756668 <== mon.2 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad000e040 con 0x7f2ae00fbaf0 2023-06-22T17:24:04.469 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.464+0000 7f2ac7fff700 1 -- 172.21.15.101:0/1029756668 <== mon.2 v2:172.21.15.101:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f2ad00095f0 con 0x7f2ae00fbaf0 2023-06-22T17:24:04.469 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.464+0000 7f2ac7fff700 1 -- 172.21.15.101:0/1029756668 <== mon.2 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad001c680 con 0x7f2ae00fbaf0 2023-06-22T17:24:04.470 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.464+0000 7f2ae5dd6700 1 -- 172.21.15.101:0/1029756668 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ae0115c00 con 0x7f2ae00fbaf0 2023-06-22T17:24:04.470 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.464+0000 7f2ae5dd6700 1 -- 172.21.15.101:0/1029756668 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ae0116010 con 0x7f2ae00fbaf0 2023-06-22T17:24:04.470 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.465+0000 7f2ae5dd6700 1 -- 172.21.15.101:0/1029756668 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ae01074c0 con 0x7f2ae00fbaf0 2023-06-22T17:24:04.470 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.466+0000 7f2ac7fff700 1 -- 172.21.15.101:0/1029756668 <== mon.2 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f2ad0019050 con 0x7f2ae00fbaf0 2023-06-22T17:24:04.471 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.466+0000 7f2ac7fff700 1 --2- 172.21.15.101:0/1029756668 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f2ac809b680 0x7f2ac809db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:24:04.471 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.466+0000 7f2ac7fff700 1 -- 172.21.15.101:0/1029756668 <== mon.2 v2:172.21.15.101:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 4961+0+0 (secure 0 0 0) 0x7f2ad00c6070 con 0x7f2ae00fbaf0 2023-06-22T17:24:04.471 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.466+0000 7f2adeffd700 1 --2- 172.21.15.101:0/1029756668 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f2ac809b680 0x7f2ac809db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:24:04.471 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.467+0000 7f2adeffd700 1 --2- 172.21.15.101:0/1029756668 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f2ac809b680 0x7f2ac809db30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2ad4009b10 tx=0x7f2ad4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:24:04.471 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.469+0000 7f2ac7fff700 1 -- 172.21.15.101:0/1029756668 <== mon.2 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f2ad00925e0 con 0x7f2ae00fbaf0 2023-06-22T17:24:04.662 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:04.660+0000 7f2ae5dd6700 1 -- 172.21.15.101:0/1029756668 --> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ae005de70 con 0x7f2ac809b680 2023-06-22T17:24:05.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:04 smithi101 ceph-mon[85507]: osdmap e46: 7 total, 7 up, 7 in 2023-06-22T17:24:05.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:04 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:24:05.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:04 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:24:05.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:04 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:05.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:04 smithi133 ceph-mon[86268]: osdmap e46: 7 total, 7 up, 7 in 2023-06-22T17:24:05.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:04 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:24:05.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:04 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:24:05.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:04 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:05.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:04 smithi039 ceph-mon[84805]: osdmap e46: 7 total, 7 up, 7 in 2023-06-22T17:24:05.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:04 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:24:05.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:04 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:24:05.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:04 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:06.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:05 smithi101 ceph-mon[85507]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:24:06.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:05 smithi101 ceph-mon[85507]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-06-22T17:24:06.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:05 smithi133 ceph-mon[86268]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:24:06.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:05 smithi133 ceph-mon[86268]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-06-22T17:24:06.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:05 smithi039 ceph-mon[84805]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:24:06.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:05 smithi039 ceph-mon[84805]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-06-22T17:24:07.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:07 smithi101 ceph-mon[85507]: from='client.? 172.21.15.101:0/2707070456' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a1cbc1b-8cf0-4153-b8e2-3d3313bed2ff"}]: dispatch 2023-06-22T17:24:07.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:07 smithi101 ceph-mon[85507]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a1cbc1b-8cf0-4153-b8e2-3d3313bed2ff"}]: dispatch 2023-06-22T17:24:07.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:07 smithi101 ceph-mon[85507]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a1cbc1b-8cf0-4153-b8e2-3d3313bed2ff"}]': finished 2023-06-22T17:24:07.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:07 smithi101 ceph-mon[85507]: osdmap e47: 8 total, 7 up, 8 in 2023-06-22T17:24:07.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:07 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-06-22T17:24:07.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:07 smithi133 ceph-mon[86268]: from='client.? 172.21.15.101:0/2707070456' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a1cbc1b-8cf0-4153-b8e2-3d3313bed2ff"}]: dispatch 2023-06-22T17:24:07.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:07 smithi133 ceph-mon[86268]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a1cbc1b-8cf0-4153-b8e2-3d3313bed2ff"}]: dispatch 2023-06-22T17:24:07.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:07 smithi133 ceph-mon[86268]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a1cbc1b-8cf0-4153-b8e2-3d3313bed2ff"}]': finished 2023-06-22T17:24:07.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:07 smithi133 ceph-mon[86268]: osdmap e47: 8 total, 7 up, 8 in 2023-06-22T17:24:07.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:07 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-06-22T17:24:07.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:07 smithi039 ceph-mon[84805]: from='client.? 172.21.15.101:0/2707070456' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a1cbc1b-8cf0-4153-b8e2-3d3313bed2ff"}]: dispatch 2023-06-22T17:24:07.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:07 smithi039 ceph-mon[84805]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a1cbc1b-8cf0-4153-b8e2-3d3313bed2ff"}]: dispatch 2023-06-22T17:24:07.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:07 smithi039 ceph-mon[84805]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a1cbc1b-8cf0-4153-b8e2-3d3313bed2ff"}]': finished 2023-06-22T17:24:07.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:07 smithi039 ceph-mon[84805]: osdmap e47: 8 total, 7 up, 8 in 2023-06-22T17:24:07.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:07 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-06-22T17:24:08.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:08 smithi101 ceph-mon[85507]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-06-22T17:24:08.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:08 smithi101 ceph-mon[85507]: from='client.? 172.21.15.101:0/183527757' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:24:08.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:08 smithi133 ceph-mon[86268]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-06-22T17:24:08.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:08 smithi133 ceph-mon[86268]: from='client.? 172.21.15.101:0/183527757' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:24:08.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:08 smithi039 ceph-mon[84805]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-06-22T17:24:08.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:08 smithi039 ceph-mon[84805]: from='client.? 172.21.15.101:0/183527757' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:24:10.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:10 smithi101 ceph-mon[85507]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-06-22T17:24:10.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:10 smithi133 ceph-mon[86268]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-06-22T17:24:10.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:10 smithi039 ceph-mon[84805]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-06-22T17:24:12.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:12 smithi101 ceph-mon[85507]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-06-22T17:24:12.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:12 smithi133 ceph-mon[86268]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-06-22T17:24:12.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:12 smithi039 ceph-mon[84805]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-06-22T17:24:14.576 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:14 smithi101 ceph-mon[85507]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-06-22T17:24:14.576 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:14 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-06-22T17:24:14.576 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:14 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:14.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:14 smithi133 ceph-mon[86268]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-06-22T17:24:14.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:14 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-06-22T17:24:14.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:14 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:14.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:14 smithi039 ceph-mon[84805]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-06-22T17:24:14.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:14 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-06-22T17:24:14.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:14 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:15.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:15 smithi101 ceph-mon[85507]: Deploying daemon osd.7 on smithi101 2023-06-22T17:24:15.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:15 smithi133 ceph-mon[86268]: Deploying daemon osd.7 on smithi101 2023-06-22T17:24:15.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:15 smithi039 ceph-mon[84805]: Deploying daemon osd.7 on smithi101 2023-06-22T17:24:16.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:16 smithi101 ceph-mon[85507]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-06-22T17:24:16.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:16 smithi133 ceph-mon[86268]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-06-22T17:24:16.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:16 smithi039 ceph-mon[84805]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-06-22T17:24:18.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:18 smithi101 ceph-mon[85507]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-06-22T17:24:18.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:18 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:18.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:18 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:18.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:18 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:24:18.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:18 smithi133 ceph-mon[86268]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-06-22T17:24:18.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:18 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:18.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:18 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:18.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:18 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:24:18.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:18 smithi039 ceph-mon[84805]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-06-22T17:24:18.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:18 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:18.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:18 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:18.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:18 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:24:20.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:20 smithi101 ceph-mon[85507]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-06-22T17:24:20.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:20 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:20.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:20 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:20.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:20 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:20.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:20 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:24:20.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:20 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:20.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:20 smithi133 ceph-mon[86268]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-06-22T17:24:20.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:20 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:20.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:20 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:20.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:20 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:20.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:20 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:24:20.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:20 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:20.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:20 smithi039 ceph-mon[84805]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-06-22T17:24:20.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:20 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:20.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:20 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:20.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:20 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:20.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:20 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:24:20.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:20 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:20.774 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:20.773+0000 7f2ac7fff700 1 -- 172.21.15.101:0/1029756668 <== mgr.14150 v2:172.21.15.39:6800/886078282 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2ae005de70 con 0x7f2ac809b680 2023-06-22T17:24:20.775 INFO:teuthology.orchestra.run.smithi101.stdout:Created osd(s) 7 on host 'smithi101' 2023-06-22T17:24:20.777 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:20.776+0000 7f2ae5dd6700 1 -- 172.21.15.101:0/1029756668 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f2ac809b680 msgr2=0x7f2ac809db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:24:20.777 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:20.776+0000 7f2ae5dd6700 1 --2- 172.21.15.101:0/1029756668 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f2ac809b680 0x7f2ac809db30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2ad4009b10 tx=0x7f2ad4008040 comp rx=0 tx=0).stop 2023-06-22T17:24:20.777 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:20.776+0000 7f2ae5dd6700 1 -- 172.21.15.101:0/1029756668 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2ae00fbaf0 msgr2=0x7f2ae0065f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:24:20.777 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:20.776+0000 7f2ae5dd6700 1 --2- 172.21.15.101:0/1029756668 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2ae00fbaf0 0x7f2ae0065f00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2ad0007c50 tx=0x7f2ad0004d10 comp rx=0 tx=0).stop 2023-06-22T17:24:20.777 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:20.776+0000 7f2ae5dd6700 1 -- 172.21.15.101:0/1029756668 shutdown_connections 2023-06-22T17:24:20.777 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:20.776+0000 7f2ae5dd6700 1 --2- 172.21.15.101:0/1029756668 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f2ac809b680 0x7f2ac809db30 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:24:20.777 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:20.776+0000 7f2ae5dd6700 1 --2- 172.21.15.101:0/1029756668 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ae00fd750 0x7f2ae0114eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:24:20.778 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:20.776+0000 7f2ae5dd6700 1 --2- 172.21.15.101:0/1029756668 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2ae00fbaf0 0x7f2ae0065f00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:24:20.778 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:20.776+0000 7f2ae5dd6700 1 --2- 172.21.15.101:0/1029756668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ae00fcda0 0x7f2ae0112870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:24:20.778 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:20.776+0000 7f2ae5dd6700 1 -- 172.21.15.101:0/1029756668 >> 172.21.15.101:0/1029756668 conn(0x7f2ae00f7210 msgr2=0x7f2ae00f94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:24:20.778 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:20.776+0000 7f2ae5dd6700 1 -- 172.21.15.101:0/1029756668 shutdown_connections 2023-06-22T17:24:20.778 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:24:20.776+0000 7f2ae5dd6700 1 -- 172.21.15.101:0/1029756668 wait complete. 2023-06-22T17:24:21.223 DEBUG:teuthology.orchestra.run.smithi101:osd.7> sudo journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.7.service 2023-06-22T17:24:21.226 INFO:tasks.cephadm:Deploying osd.8 on smithi133 with /dev/vg_nvme/lv_2... 2023-06-22T17:24:21.226 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-06-22T17:24:21.255 INFO:journalctl@ceph.osd.7.smithi101.stdout:-- Logs begin at Thu 2023-06-22 17:06:34 UTC. -- 2023-06-22T17:24:21.828 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.c/config 2023-06-22T17:24:22.042 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:21 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:22.042 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:21 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:22.042 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:21 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:24:22.043 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:21 smithi133 ceph-mon[86268]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-06-22T17:24:22.049 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:24:21 smithi101 conmon[105890]: 2023-06-22T17:24:21.906+0000 7f271a83d540 -1 osd.7 0 log_to_monitors true 2023-06-22T17:24:22.050 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:21 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:22.050 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:21 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:22.050 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:21 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:24:22.050 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:21 smithi101 ceph-mon[85507]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-06-22T17:24:22.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:21 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:22.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:21 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:22.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:21 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:24:22.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:21 smithi039 ceph-mon[84805]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-06-22T17:24:23.033 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:22 smithi101 ceph-mon[85507]: from='osd.7 [v2:172.21.15.101:6816/1606613660,v1:172.21.15.101:6817/1606613660]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-06-22T17:24:23.033 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:22 smithi101 ceph-mon[85507]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-06-22T17:24:23.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:22 smithi133 ceph-mon[86268]: from='osd.7 [v2:172.21.15.101:6816/1606613660,v1:172.21.15.101:6817/1606613660]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-06-22T17:24:23.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:22 smithi133 ceph-mon[86268]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-06-22T17:24:23.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:22 smithi039 ceph-mon[84805]: from='osd.7 [v2:172.21.15.101:6816/1606613660,v1:172.21.15.101:6817/1606613660]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-06-22T17:24:23.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:22 smithi039 ceph-mon[84805]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-06-22T17:24:23.370 INFO:teuthology.orchestra.run.smithi133.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-06-22T17:24:23.371 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-06-22T17:24:23.371 INFO:teuthology.orchestra.run.smithi133.stderr: stderr: 10+0 records in 2023-06-22T17:24:23.371 INFO:teuthology.orchestra.run.smithi133.stderr:10+0 records out 2023-06-22T17:24:23.371 INFO:teuthology.orchestra.run.smithi133.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0116768 s, 898 MB/s 2023-06-22T17:24:23.371 INFO:teuthology.orchestra.run.smithi133.stderr:--> Zapping successful for: 2023-06-22T17:24:23.758 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph orch daemon add osd smithi133:vg_nvme/lv_2 2023-06-22T17:24:24.126 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:24:23 smithi101 conmon[105890]: 2023-06-22T17:24:23.769+0000 7f2711022700 -1 osd.7 0 waiting for initial osdmap 2023-06-22T17:24:24.126 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:24:23 smithi101 conmon[105890]: 2023-06-22T17:24:23.780+0000 7f270be1d700 -1 osd.7 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-22T17:24:24.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:23 smithi101 ceph-mon[85507]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-06-22T17:24:24.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:23 smithi101 ceph-mon[85507]: osdmap e48: 8 total, 7 up, 8 in 2023-06-22T17:24:24.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:23 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-06-22T17:24:24.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:23 smithi101 ceph-mon[85507]: from='osd.7 [v2:172.21.15.101:6816/1606613660,v1:172.21.15.101:6817/1606613660]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-06-22T17:24:24.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:23 smithi101 ceph-mon[85507]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-06-22T17:24:24.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:23 smithi101 ceph-mon[85507]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-06-22T17:24:24.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:23 smithi101 ceph-mon[85507]: Detected new or changed devices on smithi101 2023-06-22T17:24:24.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:23 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:24.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:23 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:24.128 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:23 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:24:24.128 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:23 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:24:24.128 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:23 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:24:24.128 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:23 smithi101 ceph-mon[85507]: Adjusting osd_memory_target on smithi101 to 5722M 2023-06-22T17:24:24.128 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:23 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:24.128 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:23 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:24.128 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:23 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:24:24.129 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:23 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:24.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:23 smithi133 ceph-mon[86268]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-06-22T17:24:24.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:23 smithi133 ceph-mon[86268]: osdmap e48: 8 total, 7 up, 8 in 2023-06-22T17:24:24.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:23 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-06-22T17:24:24.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:23 smithi133 ceph-mon[86268]: from='osd.7 [v2:172.21.15.101:6816/1606613660,v1:172.21.15.101:6817/1606613660]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-06-22T17:24:24.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:23 smithi133 ceph-mon[86268]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-06-22T17:24:24.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:23 smithi133 ceph-mon[86268]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-06-22T17:24:24.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:23 smithi133 ceph-mon[86268]: Detected new or changed devices on smithi101 2023-06-22T17:24:24.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:23 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:24.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:23 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:24.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:23 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:24:24.172 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:23 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:24:24.172 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:23 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:24:24.172 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:23 smithi133 ceph-mon[86268]: Adjusting osd_memory_target on smithi101 to 5722M 2023-06-22T17:24:24.172 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:23 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:24.172 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:23 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:24.172 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:23 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:24:24.172 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:23 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:24.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:23 smithi039 ceph-mon[84805]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-06-22T17:24:24.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:23 smithi039 ceph-mon[84805]: osdmap e48: 8 total, 7 up, 8 in 2023-06-22T17:24:24.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:23 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-06-22T17:24:24.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:23 smithi039 ceph-mon[84805]: from='osd.7 [v2:172.21.15.101:6816/1606613660,v1:172.21.15.101:6817/1606613660]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-06-22T17:24:24.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:23 smithi039 ceph-mon[84805]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-06-22T17:24:24.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:23 smithi039 ceph-mon[84805]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-06-22T17:24:24.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:23 smithi039 ceph-mon[84805]: Detected new or changed devices on smithi101 2023-06-22T17:24:24.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:23 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:24.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:23 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:24.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:23 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:24:24.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:23 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:24:24.185 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:23 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:24:24.185 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:23 smithi039 ceph-mon[84805]: Adjusting osd_memory_target on smithi101 to 5722M 2023-06-22T17:24:24.185 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:23 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:24.185 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:23 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:24.185 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:23 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:24:24.185 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:23 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:24.340 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.c/config 2023-06-22T17:24:24.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:24 smithi133 ceph-mon[86268]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2023-06-22T17:24:24.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:24 smithi133 ceph-mon[86268]: osdmap e49: 8 total, 7 up, 8 in 2023-06-22T17:24:24.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:24 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-06-22T17:24:24.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:24 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-06-22T17:24:25.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:24 smithi101 ceph-mon[85507]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2023-06-22T17:24:25.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:24 smithi101 ceph-mon[85507]: osdmap e49: 8 total, 7 up, 8 in 2023-06-22T17:24:25.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:24 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-06-22T17:24:25.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:24 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-06-22T17:24:25.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:24 smithi039 ceph-mon[84805]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2023-06-22T17:24:25.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:24 smithi039 ceph-mon[84805]: osdmap e49: 8 total, 7 up, 8 in 2023-06-22T17:24:25.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:24 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-06-22T17:24:25.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:24 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-06-22T17:24:25.605 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.602+0000 7f8e03578700 1 -- 172.21.15.133:0/582639551 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8dfc0fde90 msgr2=0x7f8dfc0fe2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:24:25.606 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.602+0000 7f8e03578700 1 --2- 172.21.15.133:0/582639551 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8dfc0fde90 0x7f8dfc0fe2e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8dec0049c0 tx=0x7f8dec009e30 comp rx=0 tx=0).stop 2023-06-22T17:24:25.606 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.602+0000 7f8e03578700 1 -- 172.21.15.133:0/582639551 shutdown_connections 2023-06-22T17:24:25.606 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.602+0000 7f8e03578700 1 --2- 172.21.15.133:0/582639551 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8dfc0fe820 0x7f8dfc1050e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:24:25.606 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.602+0000 7f8e03578700 1 --2- 172.21.15.133:0/582639551 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8dfc0fcc80 0x7f8dfc0fd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:24:25.606 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.602+0000 7f8e03578700 1 --2- 172.21.15.133:0/582639551 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8dfc0fde90 0x7f8dfc0fe2e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:24:25.606 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.602+0000 7f8e03578700 1 -- 172.21.15.133:0/582639551 >> 172.21.15.133:0/582639551 conn(0x7f8dfc0f83c0 msgr2=0x7f8dfc0fa810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:24:25.606 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.602+0000 7f8e03578700 1 -- 172.21.15.133:0/582639551 shutdown_connections 2023-06-22T17:24:25.607 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.602+0000 7f8e03578700 1 -- 172.21.15.133:0/582639551 wait complete. 2023-06-22T17:24:25.607 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.602+0000 7f8e03578700 1 Processor -- start 2023-06-22T17:24:25.607 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.603+0000 7f8e03578700 1 -- start start 2023-06-22T17:24:25.607 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.603+0000 7f8e03578700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8dfc0fcc80 0x7f8dfc10d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:24:25.607 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.603+0000 7f8e03578700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8dfc0fde90 0x7f8dfc10dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:24:25.607 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.603+0000 7f8e03578700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8dfc0fe820 0x7f8dfc114f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:24:25.607 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.603+0000 7f8e03578700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc115540 con 0x7f8dfc0fcc80 2023-06-22T17:24:25.607 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.603+0000 7f8e03578700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc1156b0 con 0x7f8dfc0fe820 2023-06-22T17:24:25.608 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.603+0000 7f8e03578700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc115870 con 0x7f8dfc0fde90 2023-06-22T17:24:25.608 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.603+0000 7f8e00b13700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8dfc0fde90 0x7f8dfc10dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:24:25.608 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.603+0000 7f8e00b13700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8dfc0fde90 0x7f8dfc10dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:41006/0 (socket says 172.21.15.133:41006) 2023-06-22T17:24:25.608 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.603+0000 7f8e00b13700 1 -- 172.21.15.133:0/2315953717 learned_addr learned my addr 172.21.15.133:0/2315953717 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-06-22T17:24:25.608 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.603+0000 7f8e01314700 1 --2- 172.21.15.133:0/2315953717 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8dfc0fcc80 0x7f8dfc10d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:24:25.608 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.603+0000 7f8e01b15700 1 --2- 172.21.15.133:0/2315953717 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8dfc0fe820 0x7f8dfc114f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:24:25.608 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.603+0000 7f8e00b13700 1 -- 172.21.15.133:0/2315953717 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8dfc0fe820 msgr2=0x7f8dfc114f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:24:25.609 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.603+0000 7f8e00b13700 1 --2- 172.21.15.133:0/2315953717 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8dfc0fe820 0x7f8dfc114f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:24:25.609 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.603+0000 7f8e00b13700 1 -- 172.21.15.133:0/2315953717 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8dfc0fcc80 msgr2=0x7f8dfc10d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:24:25.609 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.603+0000 7f8e00b13700 1 --2- 172.21.15.133:0/2315953717 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8dfc0fcc80 0x7f8dfc10d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:24:25.609 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.603+0000 7f8e00b13700 1 -- 172.21.15.133:0/2315953717 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dec005040 con 0x7f8dfc0fde90 2023-06-22T17:24:25.609 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.603+0000 7f8e01314700 1 --2- 172.21.15.133:0/2315953717 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8dfc0fcc80 0x7f8dfc10d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T17:24:25.609 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.603+0000 7f8e01b15700 1 --2- 172.21.15.133:0/2315953717 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8dfc0fe820 0x7f8dfc114f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T17:24:25.609 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.603+0000 7f8e00b13700 1 --2- 172.21.15.133:0/2315953717 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8dfc0fde90 0x7f8dfc10dea0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8dec00bf20 tx=0x7f8dec009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:24:25.609 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.603+0000 7f8df1ffb700 1 -- 172.21.15.133:0/2315953717 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dec00e040 con 0x7f8dfc0fde90 2023-06-22T17:24:25.610 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.603+0000 7f8e03578700 1 -- 172.21.15.133:0/2315953717 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dfc115af0 con 0x7f8dfc0fde90 2023-06-22T17:24:25.610 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.603+0000 7f8df1ffb700 1 -- 172.21.15.133:0/2315953717 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f8dec013420 con 0x7f8dfc0fde90 2023-06-22T17:24:25.610 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.603+0000 7f8df1ffb700 1 -- 172.21.15.133:0/2315953717 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dec01c660 con 0x7f8dfc0fde90 2023-06-22T17:24:25.610 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.604+0000 7f8e03578700 1 -- 172.21.15.133:0/2315953717 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dfc116020 con 0x7f8dfc0fde90 2023-06-22T17:24:25.611 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.605+0000 7f8e03578700 1 -- 172.21.15.133:0/2315953717 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dfc104100 con 0x7f8dfc0fde90 2023-06-22T17:24:25.611 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.605+0000 7f8df1ffb700 1 -- 172.21.15.133:0/2315953717 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f8dec013590 con 0x7f8dfc0fde90 2023-06-22T17:24:25.611 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.606+0000 7f8df1ffb700 1 --2- 172.21.15.133:0/2315953717 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f8de809b700 0x7f8de809dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:24:25.611 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.606+0000 7f8df1ffb700 1 -- 172.21.15.133:0/2315953717 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5426+0+0 (secure 0 0 0) 0x7f8dec0c5650 con 0x7f8dfc0fde90 2023-06-22T17:24:25.611 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.606+0000 7f8e01314700 1 --2- 172.21.15.133:0/2315953717 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f8de809b700 0x7f8de809dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:24:25.611 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.607+0000 7f8e01314700 1 --2- 172.21.15.133:0/2315953717 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f8de809b700 0x7f8de809dbb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc0fdcf0 tx=0x7f8df8009040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:24:25.612 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.610+0000 7f8df1ffb700 1 -- 172.21.15.133:0/2315953717 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f8dec091950 con 0x7f8dfc0fde90 2023-06-22T17:24:25.803 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:25.801+0000 7f8e03578700 1 -- 172.21.15.133:0/2315953717 --> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f8dfc02b0d0 con 0x7f8de809b700 2023-06-22T17:24:26.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:25 smithi101 ceph-mon[85507]: purged_snaps scrub starts 2023-06-22T17:24:26.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:25 smithi101 ceph-mon[85507]: purged_snaps scrub ok 2023-06-22T17:24:26.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:25 smithi101 ceph-mon[85507]: osd.7 [v2:172.21.15.101:6816/1606613660,v1:172.21.15.101:6817/1606613660] boot 2023-06-22T17:24:26.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:25 smithi101 ceph-mon[85507]: osdmap e50: 8 total, 8 up, 8 in 2023-06-22T17:24:26.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:25 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-06-22T17:24:26.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:25 smithi101 ceph-mon[85507]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-06-22T17:24:26.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:25 smithi133 ceph-mon[86268]: purged_snaps scrub starts 2023-06-22T17:24:26.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:25 smithi133 ceph-mon[86268]: purged_snaps scrub ok 2023-06-22T17:24:26.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:25 smithi133 ceph-mon[86268]: osd.7 [v2:172.21.15.101:6816/1606613660,v1:172.21.15.101:6817/1606613660] boot 2023-06-22T17:24:26.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:25 smithi133 ceph-mon[86268]: osdmap e50: 8 total, 8 up, 8 in 2023-06-22T17:24:26.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:25 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-06-22T17:24:26.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:25 smithi133 ceph-mon[86268]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-06-22T17:24:26.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:25 smithi039 ceph-mon[84805]: purged_snaps scrub starts 2023-06-22T17:24:26.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:25 smithi039 ceph-mon[84805]: purged_snaps scrub ok 2023-06-22T17:24:26.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:25 smithi039 ceph-mon[84805]: osd.7 [v2:172.21.15.101:6816/1606613660,v1:172.21.15.101:6817/1606613660] boot 2023-06-22T17:24:26.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:25 smithi039 ceph-mon[84805]: osdmap e50: 8 total, 8 up, 8 in 2023-06-22T17:24:26.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:25 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-06-22T17:24:26.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:25 smithi039 ceph-mon[84805]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-06-22T17:24:26.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:26 smithi133 ceph-mon[86268]: osdmap e51: 8 total, 8 up, 8 in 2023-06-22T17:24:26.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:26 smithi133 ceph-mon[86268]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:24:26.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:26 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:24:26.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:26 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:24:26.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:26 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:26.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:26 smithi133 ceph-mon[86268]: osdmap e52: 8 total, 8 up, 8 in 2023-06-22T17:24:27.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:26 smithi101 ceph-mon[85507]: osdmap e51: 8 total, 8 up, 8 in 2023-06-22T17:24:27.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:26 smithi101 ceph-mon[85507]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:24:27.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:26 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:24:27.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:26 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:24:27.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:26 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:27.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:26 smithi101 ceph-mon[85507]: osdmap e52: 8 total, 8 up, 8 in 2023-06-22T17:24:27.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:26 smithi039 ceph-mon[84805]: osdmap e51: 8 total, 8 up, 8 in 2023-06-22T17:24:27.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:26 smithi039 ceph-mon[84805]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:24:27.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:26 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:24:27.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:26 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:24:27.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:26 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:27.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:26 smithi039 ceph-mon[84805]: osdmap e52: 8 total, 8 up, 8 in 2023-06-22T17:24:28.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:27 smithi101 ceph-mon[85507]: pgmap v162: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-06-22T17:24:28.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:27 smithi101 ceph-mon[85507]: osdmap e53: 8 total, 8 up, 8 in 2023-06-22T17:24:28.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:27 smithi133 ceph-mon[86268]: pgmap v162: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-06-22T17:24:28.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:27 smithi133 ceph-mon[86268]: osdmap e53: 8 total, 8 up, 8 in 2023-06-22T17:24:28.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:27 smithi039 ceph-mon[84805]: pgmap v162: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-06-22T17:24:28.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:27 smithi039 ceph-mon[84805]: osdmap e53: 8 total, 8 up, 8 in 2023-06-22T17:24:29.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:28 smithi101 ceph-mon[85507]: from='client.? 172.21.15.133:0/186946368' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab09136e-4326-4c02-ac59-3d2264e85b8b"}]: dispatch 2023-06-22T17:24:29.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:28 smithi101 ceph-mon[85507]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab09136e-4326-4c02-ac59-3d2264e85b8b"}]: dispatch 2023-06-22T17:24:29.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:28 smithi101 ceph-mon[85507]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ab09136e-4326-4c02-ac59-3d2264e85b8b"}]': finished 2023-06-22T17:24:29.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:28 smithi101 ceph-mon[85507]: osdmap e54: 9 total, 8 up, 9 in 2023-06-22T17:24:29.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:28 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-06-22T17:24:29.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:28 smithi133 ceph-mon[86268]: from='client.? 172.21.15.133:0/186946368' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab09136e-4326-4c02-ac59-3d2264e85b8b"}]: dispatch 2023-06-22T17:24:29.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:28 smithi133 ceph-mon[86268]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab09136e-4326-4c02-ac59-3d2264e85b8b"}]: dispatch 2023-06-22T17:24:29.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:28 smithi133 ceph-mon[86268]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ab09136e-4326-4c02-ac59-3d2264e85b8b"}]': finished 2023-06-22T17:24:29.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:28 smithi133 ceph-mon[86268]: osdmap e54: 9 total, 8 up, 9 in 2023-06-22T17:24:29.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:28 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-06-22T17:24:29.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:28 smithi039 ceph-mon[84805]: from='client.? 172.21.15.133:0/186946368' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab09136e-4326-4c02-ac59-3d2264e85b8b"}]: dispatch 2023-06-22T17:24:29.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:28 smithi039 ceph-mon[84805]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab09136e-4326-4c02-ac59-3d2264e85b8b"}]: dispatch 2023-06-22T17:24:29.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:28 smithi039 ceph-mon[84805]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ab09136e-4326-4c02-ac59-3d2264e85b8b"}]': finished 2023-06-22T17:24:29.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:28 smithi039 ceph-mon[84805]: osdmap e54: 9 total, 8 up, 9 in 2023-06-22T17:24:29.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:28 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-06-22T17:24:30.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:29 smithi101 ceph-mon[85507]: from='client.? 172.21.15.133:0/3887512054' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:24:30.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:29 smithi101 ceph-mon[85507]: pgmap v165: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-06-22T17:24:30.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:29 smithi133 ceph-mon[86268]: from='client.? 172.21.15.133:0/3887512054' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:24:30.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:29 smithi133 ceph-mon[86268]: pgmap v165: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-06-22T17:24:30.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:29 smithi039 ceph-mon[84805]: from='client.? 172.21.15.133:0/3887512054' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:24:30.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:29 smithi039 ceph-mon[84805]: pgmap v165: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-06-22T17:24:32.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:32 smithi133 ceph-mon[86268]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-06-22T17:24:32.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:32 smithi101 ceph-mon[85507]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-06-22T17:24:32.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:32 smithi039 ceph-mon[84805]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-06-22T17:24:34.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:34 smithi133 ceph-mon[86268]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-06-22T17:24:34.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:34 smithi101 ceph-mon[85507]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-06-22T17:24:34.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:34 smithi039 ceph-mon[84805]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-06-22T17:24:35.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:35 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-06-22T17:24:35.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:35 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:35.663 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:35 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-06-22T17:24:35.664 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:35 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:35.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:35 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-06-22T17:24:35.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:35 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:36.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:36 smithi101 ceph-mon[85507]: Deploying daemon osd.8 on smithi133 2023-06-22T17:24:36.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:36 smithi101 ceph-mon[85507]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-06-22T17:24:36.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:36 smithi133 ceph-mon[86268]: Deploying daemon osd.8 on smithi133 2023-06-22T17:24:36.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:36 smithi133 ceph-mon[86268]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-06-22T17:24:36.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:36 smithi039 ceph-mon[84805]: Deploying daemon osd.8 on smithi133 2023-06-22T17:24:36.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:36 smithi039 ceph-mon[84805]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-06-22T17:24:38.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:38 smithi101 ceph-mon[85507]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-06-22T17:24:38.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:38 smithi133 ceph-mon[86268]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-06-22T17:24:38.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:38 smithi039 ceph-mon[84805]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-06-22T17:24:40.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:40 smithi133 ceph-mon[86268]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-06-22T17:24:40.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:40 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:40.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:40 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:40.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:40 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:24:40.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:40 smithi101 ceph-mon[85507]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-06-22T17:24:40.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:40 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:40.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:40 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:40.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:40 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:24:40.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:40 smithi039 ceph-mon[84805]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-06-22T17:24:40.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:40 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:40.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:40 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:40.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:40 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:24:42.216 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:42.214+0000 7f8df1ffb700 1 -- 172.21.15.133:0/2315953717 <== mgr.14150 v2:172.21.15.39:6800/886078282 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8dfc02b0d0 con 0x7f8de809b700 2023-06-22T17:24:42.216 INFO:teuthology.orchestra.run.smithi133.stdout:Created osd(s) 8 on host 'smithi133' 2023-06-22T17:24:42.218 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:42.217+0000 7f8e03578700 1 -- 172.21.15.133:0/2315953717 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f8de809b700 msgr2=0x7f8de809dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:24:42.218 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:42.217+0000 7f8e03578700 1 --2- 172.21.15.133:0/2315953717 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f8de809b700 0x7f8de809dbb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc0fdcf0 tx=0x7f8df8009040 comp rx=0 tx=0).stop 2023-06-22T17:24:42.218 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:42.217+0000 7f8e03578700 1 -- 172.21.15.133:0/2315953717 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8dfc0fde90 msgr2=0x7f8dfc10dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:24:42.218 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:42.217+0000 7f8e03578700 1 --2- 172.21.15.133:0/2315953717 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8dfc0fde90 0x7f8dfc10dea0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8dec00bf20 tx=0x7f8dec009e30 comp rx=0 tx=0).stop 2023-06-22T17:24:42.219 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:42.217+0000 7f8e03578700 1 -- 172.21.15.133:0/2315953717 shutdown_connections 2023-06-22T17:24:42.219 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:42.217+0000 7f8e03578700 1 --2- 172.21.15.133:0/2315953717 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f8de809b700 0x7f8de809dbb0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:24:42.219 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:42.217+0000 7f8e03578700 1 --2- 172.21.15.133:0/2315953717 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8dfc0fe820 0x7f8dfc114f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:24:42.219 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:42.217+0000 7f8e03578700 1 --2- 172.21.15.133:0/2315953717 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8dfc0fcc80 0x7f8dfc10d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:24:42.219 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:42.217+0000 7f8e03578700 1 --2- 172.21.15.133:0/2315953717 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8dfc0fde90 0x7f8dfc10dea0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:24:42.219 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:42.217+0000 7f8e03578700 1 -- 172.21.15.133:0/2315953717 >> 172.21.15.133:0/2315953717 conn(0x7f8dfc0f83c0 msgr2=0x7f8dfc101280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:24:42.219 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:42.217+0000 7f8e03578700 1 -- 172.21.15.133:0/2315953717 shutdown_connections 2023-06-22T17:24:42.220 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:24:42.217+0000 7f8e03578700 1 -- 172.21.15.133:0/2315953717 wait complete. 2023-06-22T17:24:42.564 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:42 smithi133 ceph-mon[86268]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-06-22T17:24:42.564 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:42 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:42.564 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:42 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:42.565 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:42 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:42.565 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:42 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:24:42.565 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:42 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:42.565 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:42 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:42.565 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:42 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:42.565 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:42 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:24:42.588 DEBUG:teuthology.orchestra.run.smithi133:osd.8> sudo journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.8.service 2023-06-22T17:24:42.590 INFO:tasks.cephadm:Deploying osd.9 on smithi039 with /dev/vg_nvme/lv_1... 2023-06-22T17:24:42.591 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-06-22T17:24:42.618 INFO:journalctl@ceph.osd.8.smithi133.stdout:-- Logs begin at Thu 2023-06-22 17:06:34 UTC. -- 2023-06-22T17:24:42.623 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:42 smithi039 ceph-mon[84805]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-06-22T17:24:42.623 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:42 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:42.623 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:42 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:42.623 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:42 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:42.623 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:42 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:24:42.624 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:42 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:42.624 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:42 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:42.624 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:42 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:42.624 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:42 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:24:42.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:42 smithi101 ceph-mon[85507]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-06-22T17:24:42.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:42 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:42.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:42 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:42.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:42 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:42.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:42 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:24:42.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:42 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:42.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:42 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:42.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:42 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:42.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:42 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:24:43.203 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-22T17:24:43.755 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:24:43 smithi133 conmon[107946]: 2023-06-22T17:24:43.444+0000 7f7183dd9540 -1 osd.8 0 log_to_monitors true 2023-06-22T17:24:44.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:44 smithi101 ceph-mon[85507]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-06-22T17:24:44.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:44 smithi101 ceph-mon[85507]: from='osd.8 [v2:172.21.15.133:6816/1363381164,v1:172.21.15.133:6817/1363381164]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-06-22T17:24:44.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:44 smithi101 ceph-mon[85507]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-06-22T17:24:44.662 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:44 smithi133 ceph-mon[86268]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-06-22T17:24:44.662 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:44 smithi133 ceph-mon[86268]: from='osd.8 [v2:172.21.15.133:6816/1363381164,v1:172.21.15.133:6817/1363381164]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-06-22T17:24:44.662 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:44 smithi133 ceph-mon[86268]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-06-22T17:24:44.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:44 smithi039 ceph-mon[84805]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-06-22T17:24:44.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:44 smithi039 ceph-mon[84805]: from='osd.8 [v2:172.21.15.133:6816/1363381164,v1:172.21.15.133:6817/1363381164]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-06-22T17:24:44.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:44 smithi039 ceph-mon[84805]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-06-22T17:24:44.720 INFO:teuthology.orchestra.run.smithi039.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-06-22T17:24:44.720 INFO:teuthology.orchestra.run.smithi039.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-06-22T17:24:44.721 INFO:teuthology.orchestra.run.smithi039.stderr: stderr: 10+0 records in 2023-06-22T17:24:44.721 INFO:teuthology.orchestra.run.smithi039.stderr:10+0 records out 2023-06-22T17:24:44.721 INFO:teuthology.orchestra.run.smithi039.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0142152 s, 738 MB/s 2023-06-22T17:24:44.721 INFO:teuthology.orchestra.run.smithi039.stderr:--> Zapping successful for: 2023-06-22T17:24:45.152 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph orch daemon add osd smithi039:vg_nvme/lv_1 2023-06-22T17:24:45.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:45 smithi101 ceph-mon[85507]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-06-22T17:24:45.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:45 smithi101 ceph-mon[85507]: osdmap e55: 9 total, 8 up, 9 in 2023-06-22T17:24:45.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:45 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-06-22T17:24:45.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:45 smithi101 ceph-mon[85507]: from='osd.8 [v2:172.21.15.133:6816/1363381164,v1:172.21.15.133:6817/1363381164]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2023-06-22T17:24:45.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:45 smithi101 ceph-mon[85507]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2023-06-22T17:24:45.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:45 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:45.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:45 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:45.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:45 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:24:45.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:45 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:24:45.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:45 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:24:45.628 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:45 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:45.628 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:45 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:45.628 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:45 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:24:45.628 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:45 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:45.670 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:24:45 smithi133 conmon[107946]: 2023-06-22T17:24:45.363+0000 7f717bdc1700 -1 osd.8 0 waiting for initial osdmap 2023-06-22T17:24:45.671 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:24:45 smithi133 conmon[107946]: 2023-06-22T17:24:45.376+0000 7f71753b9700 -1 osd.8 56 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-22T17:24:45.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:45 smithi133 ceph-mon[86268]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-06-22T17:24:45.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:45 smithi133 ceph-mon[86268]: osdmap e55: 9 total, 8 up, 9 in 2023-06-22T17:24:45.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:45 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-06-22T17:24:45.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:45 smithi133 ceph-mon[86268]: from='osd.8 [v2:172.21.15.133:6816/1363381164,v1:172.21.15.133:6817/1363381164]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2023-06-22T17:24:45.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:45 smithi133 ceph-mon[86268]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2023-06-22T17:24:45.672 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:45 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:45.672 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:45 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:45.672 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:45 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:24:45.672 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:45 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:24:45.672 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:45 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:24:45.672 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:45 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:45.672 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:45 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:45.672 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:45 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:24:45.673 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:45 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:45.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:45 smithi039 ceph-mon[84805]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-06-22T17:24:45.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:45 smithi039 ceph-mon[84805]: osdmap e55: 9 total, 8 up, 9 in 2023-06-22T17:24:45.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:45 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-06-22T17:24:45.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:45 smithi039 ceph-mon[84805]: from='osd.8 [v2:172.21.15.133:6816/1363381164,v1:172.21.15.133:6817/1363381164]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2023-06-22T17:24:45.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:45 smithi039 ceph-mon[84805]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2023-06-22T17:24:45.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:45 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:45.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:45 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:45.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:45 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:24:45.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:45 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:24:45.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:45 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:24:45.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:45 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:45.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:45 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:45.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:45 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:24:45.685 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:45 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:24:45.789 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-22T17:24:46.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:46 smithi101 ceph-mon[85507]: Detected new or changed devices on smithi133 2023-06-22T17:24:46.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:46 smithi101 ceph-mon[85507]: Adjusting osd_memory_target on smithi133 to 5722M 2023-06-22T17:24:46.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:46 smithi101 ceph-mon[85507]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-06-22T17:24:46.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:46 smithi101 ceph-mon[85507]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]': finished 2023-06-22T17:24:46.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:46 smithi101 ceph-mon[85507]: osdmap e56: 9 total, 8 up, 9 in 2023-06-22T17:24:46.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:46 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-06-22T17:24:46.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:46 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-06-22T17:24:46.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:46 smithi101 ceph-mon[85507]: osd.8 [v2:172.21.15.133:6816/1363381164,v1:172.21.15.133:6817/1363381164] boot 2023-06-22T17:24:46.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:46 smithi101 ceph-mon[85507]: osdmap e57: 9 total, 9 up, 9 in 2023-06-22T17:24:46.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:46 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-06-22T17:24:46.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:46 smithi133 ceph-mon[86268]: Detected new or changed devices on smithi133 2023-06-22T17:24:46.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:46 smithi133 ceph-mon[86268]: Adjusting osd_memory_target on smithi133 to 5722M 2023-06-22T17:24:46.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:46 smithi133 ceph-mon[86268]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-06-22T17:24:46.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:46 smithi133 ceph-mon[86268]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]': finished 2023-06-22T17:24:46.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:46 smithi133 ceph-mon[86268]: osdmap e56: 9 total, 8 up, 9 in 2023-06-22T17:24:46.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:46 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-06-22T17:24:46.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:46 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-06-22T17:24:46.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:46 smithi133 ceph-mon[86268]: osd.8 [v2:172.21.15.133:6816/1363381164,v1:172.21.15.133:6817/1363381164] boot 2023-06-22T17:24:46.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:46 smithi133 ceph-mon[86268]: osdmap e57: 9 total, 9 up, 9 in 2023-06-22T17:24:46.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:46 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-06-22T17:24:46.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:46 smithi039 ceph-mon[84805]: Detected new or changed devices on smithi133 2023-06-22T17:24:46.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:46 smithi039 ceph-mon[84805]: Adjusting osd_memory_target on smithi133 to 5722M 2023-06-22T17:24:46.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:46 smithi039 ceph-mon[84805]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-06-22T17:24:46.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:46 smithi039 ceph-mon[84805]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]': finished 2023-06-22T17:24:46.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:46 smithi039 ceph-mon[84805]: osdmap e56: 9 total, 8 up, 9 in 2023-06-22T17:24:46.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:46 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-06-22T17:24:46.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:46 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-06-22T17:24:46.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:46 smithi039 ceph-mon[84805]: osd.8 [v2:172.21.15.133:6816/1363381164,v1:172.21.15.133:6817/1363381164] boot 2023-06-22T17:24:46.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:46 smithi039 ceph-mon[84805]: osdmap e57: 9 total, 9 up, 9 in 2023-06-22T17:24:46.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:46 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-06-22T17:24:47.144 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.143+0000 7f6b0b1eb700 1 -- 172.21.15.39:0/1323106299 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b040fcd20 msgr2=0x7f6b040fd170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:24:47.145 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.143+0000 7f6b0b1eb700 1 --2- 172.21.15.39:0/1323106299 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b040fcd20 0x7f6b040fd170 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6af40049c0 tx=0x7f6af4009e30 comp rx=0 tx=0).stop 2023-06-22T17:24:47.145 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.143+0000 7f6b0b1eb700 1 -- 172.21.15.39:0/1323106299 shutdown_connections 2023-06-22T17:24:47.145 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.143+0000 7f6b0b1eb700 1 --2- 172.21.15.39:0/1323106299 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6b040fd6b0 0x7f6b04103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:24:47.145 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.143+0000 7f6b0b1eb700 1 --2- 172.21.15.39:0/1323106299 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f6b040fbb10 0x7f6b040fbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:24:47.145 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.143+0000 7f6b0b1eb700 1 --2- 172.21.15.39:0/1323106299 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b040fcd20 0x7f6b040fd170 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:24:47.145 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.143+0000 7f6b0b1eb700 1 -- 172.21.15.39:0/1323106299 >> 172.21.15.39:0/1323106299 conn(0x7f6b040f7230 msgr2=0x7f6b040f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:24:47.145 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.143+0000 7f6b0b1eb700 1 -- 172.21.15.39:0/1323106299 shutdown_connections 2023-06-22T17:24:47.145 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.143+0000 7f6b0b1eb700 1 -- 172.21.15.39:0/1323106299 wait complete. 2023-06-22T17:24:47.147 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.143+0000 7f6b0b1eb700 1 Processor -- start 2023-06-22T17:24:47.147 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.143+0000 7f6b0b1eb700 1 -- start start 2023-06-22T17:24:47.147 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.144+0000 7f6b0b1eb700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f6b040fbb10 0x7f6b040ffa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:24:47.147 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.144+0000 7f6b0b1eb700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b040fcd20 0x7f6b040fe060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:24:47.148 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.144+0000 7f6b0b1eb700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6b040fd6b0 0x7f6b040fe5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:24:47.148 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.144+0000 7f6b0b1eb700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b040fed10 con 0x7f6b040fcd20 2023-06-22T17:24:47.148 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.144+0000 7f6b0b1eb700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b040fee80 con 0x7f6b040fbb10 2023-06-22T17:24:47.148 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.144+0000 7f6b0b1eb700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b04117aa0 con 0x7f6b040fd6b0 2023-06-22T17:24:47.148 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.144+0000 7f6b03fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b040fcd20 0x7f6b040fe060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:24:47.148 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.144+0000 7f6b03fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b040fcd20 0x7f6b040fe060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:42468/0 (socket says 172.21.15.39:42468) 2023-06-22T17:24:47.148 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.144+0000 7f6b03fff700 1 -- 172.21.15.39:0/3288449839 learned_addr learned my addr 172.21.15.39:0/3288449839 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:24:47.148 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.144+0000 7f6b08f87700 1 --2- 172.21.15.39:0/3288449839 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f6b040fbb10 0x7f6b040ffa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:24:47.149 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.144+0000 7f6b09788700 1 --2- 172.21.15.39:0/3288449839 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6b040fd6b0 0x7f6b040fe5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:24:47.149 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.144+0000 7f6b03fff700 1 -- 172.21.15.39:0/3288449839 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6b040fd6b0 msgr2=0x7f6b040fe5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:24:47.149 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.144+0000 7f6b03fff700 1 --2- 172.21.15.39:0/3288449839 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6b040fd6b0 0x7f6b040fe5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:24:47.149 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.144+0000 7f6b03fff700 1 -- 172.21.15.39:0/3288449839 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f6b040fbb10 msgr2=0x7f6b040ffa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:24:47.149 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.144+0000 7f6b03fff700 1 --2- 172.21.15.39:0/3288449839 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f6b040fbb10 0x7f6b040ffa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:24:47.149 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.144+0000 7f6b03fff700 1 -- 172.21.15.39:0/3288449839 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af4005040 con 0x7f6b040fcd20 2023-06-22T17:24:47.149 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.144+0000 7f6b08f87700 1 --2- 172.21.15.39:0/3288449839 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f6b040fbb10 0x7f6b040ffa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T17:24:47.150 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.144+0000 7f6b09788700 1 --2- 172.21.15.39:0/3288449839 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6b040fd6b0 0x7f6b040fe5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T17:24:47.150 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.144+0000 7f6b03fff700 1 --2- 172.21.15.39:0/3288449839 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b040fcd20 0x7f6b040fe060 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6af4000cf0 tx=0x7f6af4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:24:47.150 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.144+0000 7f6b017fa700 1 -- 172.21.15.39:0/3288449839 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af400e040 con 0x7f6b040fcd20 2023-06-22T17:24:47.150 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.144+0000 7f6b017fa700 1 -- 172.21.15.39:0/3288449839 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f6af4009830 con 0x7f6b040fcd20 2023-06-22T17:24:47.150 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.144+0000 7f6b0b1eb700 1 -- 172.21.15.39:0/3288449839 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b04117cf0 con 0x7f6b040fcd20 2023-06-22T17:24:47.150 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.144+0000 7f6b017fa700 1 -- 172.21.15.39:0/3288449839 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af401c3f0 con 0x7f6b040fcd20 2023-06-22T17:24:47.150 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.144+0000 7f6b0b1eb700 1 -- 172.21.15.39:0/3288449839 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b04118150 con 0x7f6b040fcd20 2023-06-22T17:24:47.150 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.145+0000 7f6b0b1eb700 1 -- 172.21.15.39:0/3288449839 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b04107440 con 0x7f6b040fcd20 2023-06-22T17:24:47.151 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.146+0000 7f6b017fa700 1 -- 172.21.15.39:0/3288449839 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f6af4013600 con 0x7f6b040fcd20 2023-06-22T17:24:47.151 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.147+0000 7f6b017fa700 1 --2- 172.21.15.39:0/3288449839 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f6af009b680 0x7f6af009db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:24:47.151 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.147+0000 7f6b08f87700 1 --2- 172.21.15.39:0/3288449839 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f6af009b680 0x7f6af009db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:24:47.151 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.147+0000 7f6b017fa700 1 -- 172.21.15.39:0/3288449839 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5858+0+0 (secure 0 0 0) 0x7f6af40c56d0 con 0x7f6b040fcd20 2023-06-22T17:24:47.152 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.147+0000 7f6b08f87700 1 --2- 172.21.15.39:0/3288449839 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f6af009b680 0x7f6af009db30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6aec009fd0 tx=0x7f6aec009f00 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:24:47.153 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.151+0000 7f6b017fa700 1 -- 172.21.15.39:0/3288449839 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f6af40918c0 con 0x7f6b040fcd20 2023-06-22T17:24:47.359 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:24:47.358+0000 7f6b0b1eb700 1 -- 172.21.15.39:0/3288449839 --> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b04066410 con 0x7f6af009b680 2023-06-22T17:24:47.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:47 smithi101 ceph-mon[85507]: purged_snaps scrub starts 2023-06-22T17:24:47.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:47 smithi101 ceph-mon[85507]: purged_snaps scrub ok 2023-06-22T17:24:47.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:47 smithi101 ceph-mon[85507]: osdmap e58: 9 total, 9 up, 9 in 2023-06-22T17:24:47.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:47 smithi133 ceph-mon[86268]: purged_snaps scrub starts 2023-06-22T17:24:47.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:47 smithi133 ceph-mon[86268]: purged_snaps scrub ok 2023-06-22T17:24:47.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:47 smithi133 ceph-mon[86268]: osdmap e58: 9 total, 9 up, 9 in 2023-06-22T17:24:47.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:47 smithi039 ceph-mon[84805]: purged_snaps scrub starts 2023-06-22T17:24:47.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:47 smithi039 ceph-mon[84805]: purged_snaps scrub ok 2023-06-22T17:24:47.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:47 smithi039 ceph-mon[84805]: osdmap e58: 9 total, 9 up, 9 in 2023-06-22T17:24:48.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:48 smithi039 ceph-mon[84805]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-06-22T17:24:48.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:48 smithi039 ceph-mon[84805]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:24:48.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:48 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:24:48.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:48 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:24:48.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:48 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:48.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:48 smithi039 ceph-mon[84805]: osdmap e59: 9 total, 9 up, 9 in 2023-06-22T17:24:48.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:48 smithi101 ceph-mon[85507]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-06-22T17:24:48.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:48 smithi101 ceph-mon[85507]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:24:48.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:48 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:24:48.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:48 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:24:48.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:48 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:48.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:48 smithi101 ceph-mon[85507]: osdmap e59: 9 total, 9 up, 9 in 2023-06-22T17:24:48.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:48 smithi133 ceph-mon[86268]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-06-22T17:24:48.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:48 smithi133 ceph-mon[86268]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:24:48.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:48 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:24:48.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:48 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:24:48.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:48 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:48.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:48 smithi133 ceph-mon[86268]: osdmap e59: 9 total, 9 up, 9 in 2023-06-22T17:24:50.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:50 smithi133 ceph-mon[86268]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-06-22T17:24:50.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:50 smithi133 ceph-mon[86268]: osdmap e60: 9 total, 9 up, 9 in 2023-06-22T17:24:50.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:50 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/860137074' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec8c1344-0478-4094-b544-3360054fd92b"}]: dispatch 2023-06-22T17:24:50.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:50 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/860137074' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec8c1344-0478-4094-b544-3360054fd92b"}]': finished 2023-06-22T17:24:50.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:50 smithi133 ceph-mon[86268]: osdmap e61: 10 total, 9 up, 10 in 2023-06-22T17:24:50.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:50 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-06-22T17:24:50.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:50 smithi039 ceph-mon[84805]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-06-22T17:24:50.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:50 smithi039 ceph-mon[84805]: osdmap e60: 9 total, 9 up, 9 in 2023-06-22T17:24:50.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:50 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/860137074' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec8c1344-0478-4094-b544-3360054fd92b"}]: dispatch 2023-06-22T17:24:50.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:50 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/860137074' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec8c1344-0478-4094-b544-3360054fd92b"}]': finished 2023-06-22T17:24:50.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:50 smithi039 ceph-mon[84805]: osdmap e61: 10 total, 9 up, 10 in 2023-06-22T17:24:50.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:50 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-06-22T17:24:50.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:50 smithi101 ceph-mon[85507]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-06-22T17:24:50.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:50 smithi101 ceph-mon[85507]: osdmap e60: 9 total, 9 up, 9 in 2023-06-22T17:24:50.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:50 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/860137074' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec8c1344-0478-4094-b544-3360054fd92b"}]: dispatch 2023-06-22T17:24:50.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:50 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/860137074' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec8c1344-0478-4094-b544-3360054fd92b"}]': finished 2023-06-22T17:24:50.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:50 smithi101 ceph-mon[85507]: osdmap e61: 10 total, 9 up, 10 in 2023-06-22T17:24:50.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:50 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-06-22T17:24:52.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:51 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/3710194304' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:24:52.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:51 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/3710194304' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:24:52.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:51 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/3710194304' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:24:53.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:52 smithi101 ceph-mon[85507]: pgmap v183: 1 pgs: 1 peering; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-06-22T17:24:53.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:52 smithi133 ceph-mon[86268]: pgmap v183: 1 pgs: 1 peering; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-06-22T17:24:53.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:52 smithi039 ceph-mon[84805]: pgmap v183: 1 pgs: 1 peering; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-06-22T17:24:54.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:53 smithi101 ceph-mon[85507]: pgmap v184: 1 pgs: 1 peering; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-06-22T17:24:54.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:53 smithi133 ceph-mon[86268]: pgmap v184: 1 pgs: 1 peering; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-06-22T17:24:54.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:53 smithi039 ceph-mon[84805]: pgmap v184: 1 pgs: 1 peering; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-06-22T17:24:56.638 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:56 smithi039 ceph-mon[84805]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2023-06-22T17:24:56.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:56 smithi133 ceph-mon[86268]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2023-06-22T17:24:56.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:56 smithi101 ceph-mon[85507]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2023-06-22T17:24:58.814 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:58 smithi039 ceph-mon[84805]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-06-22T17:24:58.815 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:58 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-06-22T17:24:58.815 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:58 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:58.815 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:24:58 smithi039 ceph-mon[84805]: Deploying daemon osd.9 on smithi039 2023-06-22T17:24:58.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:58 smithi101 ceph-mon[85507]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-06-22T17:24:58.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:58 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-06-22T17:24:58.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:58 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:58.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:24:58 smithi101 ceph-mon[85507]: Deploying daemon osd.9 on smithi039 2023-06-22T17:24:58.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:58 smithi133 ceph-mon[86268]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-06-22T17:24:58.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:58 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-06-22T17:24:58.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:58 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:24:58.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:24:58 smithi133 ceph-mon[86268]: Deploying daemon osd.9 on smithi039 2023-06-22T17:25:00.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:00 smithi039 ceph-mon[84805]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-06-22T17:25:00.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:00 smithi101 ceph-mon[85507]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-06-22T17:25:00.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:00 smithi133 ceph-mon[86268]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-06-22T17:25:02.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:02 smithi101 ceph-mon[85507]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-06-22T17:25:02.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:02 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:02.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:02 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:02.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:02 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:25:02.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:02 smithi133 ceph-mon[86268]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-06-22T17:25:02.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:02.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:02.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:25:02.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:02 smithi039 ceph-mon[84805]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-06-22T17:25:02.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:02 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:02.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:02 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:02.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:02 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:25:04.750 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:04 smithi039 ceph-mon[84805]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-06-22T17:25:04.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:04 smithi101 ceph-mon[85507]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-06-22T17:25:04.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:04 smithi133 ceph-mon[86268]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-06-22T17:25:04.969 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:04.967+0000 7f6b017fa700 1 -- 172.21.15.39:0/3288449839 <== mgr.14150 v2:172.21.15.39:6800/886078282 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6b04066410 con 0x7f6af009b680 2023-06-22T17:25:04.970 INFO:teuthology.orchestra.run.smithi039.stdout:Created osd(s) 9 on host 'smithi039' 2023-06-22T17:25:04.971 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:04.969+0000 7f6b0b1eb700 1 -- 172.21.15.39:0/3288449839 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f6af009b680 msgr2=0x7f6af009db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:04.971 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:04.969+0000 7f6b0b1eb700 1 --2- 172.21.15.39:0/3288449839 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f6af009b680 0x7f6af009db30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6aec009fd0 tx=0x7f6aec009f00 comp rx=0 tx=0).stop 2023-06-22T17:25:04.972 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:04.969+0000 7f6b0b1eb700 1 -- 172.21.15.39:0/3288449839 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b040fcd20 msgr2=0x7f6b040fe060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:04.972 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:04.969+0000 7f6b0b1eb700 1 --2- 172.21.15.39:0/3288449839 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b040fcd20 0x7f6b040fe060 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6af4000cf0 tx=0x7f6af4009e30 comp rx=0 tx=0).stop 2023-06-22T17:25:04.972 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:04.970+0000 7f6b0b1eb700 1 -- 172.21.15.39:0/3288449839 shutdown_connections 2023-06-22T17:25:04.972 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:04.970+0000 7f6b0b1eb700 1 --2- 172.21.15.39:0/3288449839 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f6af009b680 0x7f6af009db30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:04.972 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:04.970+0000 7f6b0b1eb700 1 --2- 172.21.15.39:0/3288449839 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6b040fd6b0 0x7f6b040fe5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:04.972 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:04.970+0000 7f6b0b1eb700 1 --2- 172.21.15.39:0/3288449839 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f6b040fbb10 0x7f6b040ffa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:04.972 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:04.970+0000 7f6b0b1eb700 1 --2- 172.21.15.39:0/3288449839 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b040fcd20 0x7f6b040fe060 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:04.973 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:04.970+0000 7f6b0b1eb700 1 -- 172.21.15.39:0/3288449839 >> 172.21.15.39:0/3288449839 conn(0x7f6b040f7230 msgr2=0x7f6b04065d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:25:04.973 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:04.970+0000 7f6b0b1eb700 1 -- 172.21.15.39:0/3288449839 shutdown_connections 2023-06-22T17:25:04.973 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:04.970+0000 7f6b0b1eb700 1 -- 172.21.15.39:0/3288449839 wait complete. 2023-06-22T17:25:05.367 DEBUG:teuthology.orchestra.run.smithi039:osd.9> sudo journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.9.service 2023-06-22T17:25:05.369 INFO:tasks.cephadm:Deploying osd.10 on smithi101 with /dev/vg_nvme/lv_1... 2023-06-22T17:25:05.370 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-06-22T17:25:05.401 INFO:journalctl@ceph.osd.9.smithi039.stdout:-- Logs begin at Thu 2023-06-22 17:07:05 UTC. -- 2023-06-22T17:25:05.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:05 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:05.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:05 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:05.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:05 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:05.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:05 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:05.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:05 smithi039 ceph-mon[84805]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-06-22T17:25:05.962 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.b/config 2023-06-22T17:25:06.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:05 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:06.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:05 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:06.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:05 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:06.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:05 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:06.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:05 smithi101 ceph-mon[85507]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-06-22T17:25:06.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:05 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:06.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:05 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:06.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:05 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:06.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:05 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:06.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:05 smithi133 ceph-mon[86268]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-06-22T17:25:06.433 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:25:06 smithi039 conmon[122618]: 2023-06-22T17:25:06.120+0000 7f7e19cb7540 -1 osd.9 0 log_to_monitors true 2023-06-22T17:25:07.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:06 smithi101 ceph-mon[85507]: from='osd.9 [v2:172.21.15.39:6826/3010986736,v1:172.21.15.39:6827/3010986736]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-06-22T17:25:07.418 INFO:teuthology.orchestra.run.smithi101.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-06-22T17:25:07.418 INFO:teuthology.orchestra.run.smithi101.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-06-22T17:25:07.418 INFO:teuthology.orchestra.run.smithi101.stderr: stderr: 10+0 records in 2023-06-22T17:25:07.418 INFO:teuthology.orchestra.run.smithi101.stderr:10+0 records out 2023-06-22T17:25:07.418 INFO:teuthology.orchestra.run.smithi101.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0115499 s, 908 MB/s 2023-06-22T17:25:07.419 INFO:teuthology.orchestra.run.smithi101.stderr: stderr: 2023-06-22T17:25:07.419 INFO:teuthology.orchestra.run.smithi101.stderr:--> Zapping successful for: 2023-06-22T17:25:07.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:06 smithi133 ceph-mon[86268]: from='osd.9 [v2:172.21.15.39:6826/3010986736,v1:172.21.15.39:6827/3010986736]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-06-22T17:25:07.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:06 smithi039 ceph-mon[84805]: from='osd.9 [v2:172.21.15.39:6826/3010986736,v1:172.21.15.39:6827/3010986736]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-06-22T17:25:07.797 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph orch daemon add osd smithi101:vg_nvme/lv_1 2023-06-22T17:25:08.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:07 smithi101 ceph-mon[85507]: from='osd.9 [v2:172.21.15.39:6826/3010986736,v1:172.21.15.39:6827/3010986736]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-06-22T17:25:08.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:07 smithi101 ceph-mon[85507]: osdmap e62: 10 total, 9 up, 10 in 2023-06-22T17:25:08.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:07 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-06-22T17:25:08.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:07 smithi101 ceph-mon[85507]: from='osd.9 [v2:172.21.15.39:6826/3010986736,v1:172.21.15.39:6827/3010986736]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-06-22T17:25:08.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:07 smithi101 ceph-mon[85507]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-06-22T17:25:08.183 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:25:07 smithi039 conmon[122618]: 2023-06-22T17:25:07.980+0000 7f7e1049c700 -1 osd.9 0 waiting for initial osdmap 2023-06-22T17:25:08.183 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:25:07 smithi039 conmon[122618]: 2023-06-22T17:25:07.996+0000 7f7e0b297700 -1 osd.9 63 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-22T17:25:08.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:07 smithi039 ceph-mon[84805]: from='osd.9 [v2:172.21.15.39:6826/3010986736,v1:172.21.15.39:6827/3010986736]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-06-22T17:25:08.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:07 smithi039 ceph-mon[84805]: osdmap e62: 10 total, 9 up, 10 in 2023-06-22T17:25:08.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:07 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-06-22T17:25:08.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:07 smithi039 ceph-mon[84805]: from='osd.9 [v2:172.21.15.39:6826/3010986736,v1:172.21.15.39:6827/3010986736]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-06-22T17:25:08.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:07 smithi039 ceph-mon[84805]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-06-22T17:25:08.378 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.b/config 2023-06-22T17:25:08.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:07 smithi133 ceph-mon[86268]: from='osd.9 [v2:172.21.15.39:6826/3010986736,v1:172.21.15.39:6827/3010986736]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-06-22T17:25:08.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:07 smithi133 ceph-mon[86268]: osdmap e62: 10 total, 9 up, 10 in 2023-06-22T17:25:08.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:07 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-06-22T17:25:08.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:07 smithi133 ceph-mon[86268]: from='osd.9 [v2:172.21.15.39:6826/3010986736,v1:172.21.15.39:6827/3010986736]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-06-22T17:25:08.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:07 smithi133 ceph-mon[86268]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-06-22T17:25:09.329 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:08 smithi101 ceph-mon[85507]: from='osd.9 [v2:172.21.15.39:6826/3010986736,v1:172.21.15.39:6827/3010986736]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2023-06-22T17:25:09.329 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:08 smithi101 ceph-mon[85507]: osdmap e63: 10 total, 9 up, 10 in 2023-06-22T17:25:09.329 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:08 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-06-22T17:25:09.330 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:08 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-06-22T17:25:09.330 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:08 smithi101 ceph-mon[85507]: Detected new or changed devices on smithi039 2023-06-22T17:25:09.330 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:08 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:09.330 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:08 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:09.330 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:08 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:09.330 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:08 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:09.330 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:08 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:09.331 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:08 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:09.331 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:08 smithi101 ceph-mon[85507]: Adjusting osd_memory_target on smithi039 to 4292M 2023-06-22T17:25:09.331 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:08 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:09.331 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:08 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:25:09.331 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:08 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:25:09.331 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:08 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:09.331 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:08 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:25:09.331 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:08 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:25:09.332 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:08 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:25:09.332 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:08 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:09.332 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:08 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-06-22T17:25:09.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:08 smithi133 ceph-mon[86268]: from='osd.9 [v2:172.21.15.39:6826/3010986736,v1:172.21.15.39:6827/3010986736]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2023-06-22T17:25:09.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:08 smithi133 ceph-mon[86268]: osdmap e63: 10 total, 9 up, 10 in 2023-06-22T17:25:09.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:08 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-06-22T17:25:09.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:08 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-06-22T17:25:09.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:08 smithi133 ceph-mon[86268]: Detected new or changed devices on smithi039 2023-06-22T17:25:09.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:08 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:09.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:08 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:09.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:08 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:09.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:08 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:09.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:08 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:09.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:08 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:09.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:08 smithi133 ceph-mon[86268]: Adjusting osd_memory_target on smithi039 to 4292M 2023-06-22T17:25:09.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:08 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:09.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:08 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:25:09.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:08 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:25:09.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:08 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:09.423 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:08 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:25:09.423 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:08 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:25:09.423 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:08 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:25:09.423 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:08 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:09.423 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:08 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-06-22T17:25:09.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:08 smithi039 ceph-mon[84805]: from='osd.9 [v2:172.21.15.39:6826/3010986736,v1:172.21.15.39:6827/3010986736]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2023-06-22T17:25:09.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:08 smithi039 ceph-mon[84805]: osdmap e63: 10 total, 9 up, 10 in 2023-06-22T17:25:09.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:08 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-06-22T17:25:09.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:08 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-06-22T17:25:09.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:08 smithi039 ceph-mon[84805]: Detected new or changed devices on smithi039 2023-06-22T17:25:09.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:08 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:09.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:08 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:09.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:08 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:09.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:08 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:09.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:08 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:09.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:08 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:09.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:08 smithi039 ceph-mon[84805]: Adjusting osd_memory_target on smithi039 to 4292M 2023-06-22T17:25:09.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:08 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:09.435 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:08 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:25:09.435 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:08 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:25:09.435 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:08 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:09.435 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:08 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:25:09.435 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:08 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:25:09.435 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:08 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:25:09.435 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:08 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:09.435 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:08 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-06-22T17:25:09.558 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.556+0000 7f66b9f84700 1 -- 172.21.15.101:0/2465315967 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f66b4102880 msgr2=0x7f66b4104d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:09.558 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.556+0000 7f66b9f84700 1 --2- 172.21.15.101:0/2465315967 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f66b4102880 0x7f66b4104d30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f66ac009d60 tx=0x7f66ac004d10 comp rx=0 tx=0).stop 2023-06-22T17:25:09.558 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.556+0000 7f66b9f84700 1 -- 172.21.15.101:0/2465315967 shutdown_connections 2023-06-22T17:25:09.558 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.556+0000 7f66b9f84700 1 --2- 172.21.15.101:0/2465315967 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f66b4102880 0x7f66b4104d30 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:09.559 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.556+0000 7f66b9f84700 1 --2- 172.21.15.101:0/2465315967 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f66b4066cc0 0x7f66b40ff7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:09.559 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.556+0000 7f66b9f84700 1 --2- 172.21.15.101:0/2465315967 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66b40ffd90 0x7f66b41021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:09.559 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.556+0000 7f66b9f84700 1 -- 172.21.15.101:0/2465315967 >> 172.21.15.101:0/2465315967 conn(0x7f66b40f7270 msgr2=0x7f66b40f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:25:09.559 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.556+0000 7f66b9f84700 1 -- 172.21.15.101:0/2465315967 shutdown_connections 2023-06-22T17:25:09.559 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.556+0000 7f66b9f84700 1 -- 172.21.15.101:0/2465315967 wait complete. 2023-06-22T17:25:09.559 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.557+0000 7f66b9f84700 1 Processor -- start 2023-06-22T17:25:09.559 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.557+0000 7f66b9f84700 1 -- start start 2023-06-22T17:25:09.561 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.557+0000 7f66b9f84700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f66b4066cc0 0x7f66b40fd060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:25:09.561 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.557+0000 7f66b9f84700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66b40ffd90 0x7f66b40fb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:25:09.561 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.557+0000 7f66b9f84700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f66b4102880 0x7f66b40fbc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:25:09.561 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.557+0000 7f66b9f84700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b4111690 con 0x7f66b40ffd90 2023-06-22T17:25:09.561 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.557+0000 7f66b9f84700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b40fc140 con 0x7f66b4066cc0 2023-06-22T17:25:09.562 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.557+0000 7f66b9f84700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b40fc2b0 con 0x7f66b4102880 2023-06-22T17:25:09.562 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.557+0000 7f66b37fe700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f66b4066cc0 0x7f66b40fd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:25:09.562 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.557+0000 7f66b37fe700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f66b4066cc0 0x7f66b40fd060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:59786/0 (socket says 172.21.15.101:59786) 2023-06-22T17:25:09.562 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.557+0000 7f66b37fe700 1 -- 172.21.15.101:0/624443025 learned_addr learned my addr 172.21.15.101:0/624443025 (peer_addr_for_me v2:172.21.15.101:0/0) 2023-06-22T17:25:09.562 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.557+0000 7f66b3fff700 1 --2- 172.21.15.101:0/624443025 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f66b4102880 0x7f66b40fbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:25:09.562 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.557+0000 7f66b2ffd700 1 --2- 172.21.15.101:0/624443025 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66b40ffd90 0x7f66b40fb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:25:09.562 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.558+0000 7f66b37fe700 1 -- 172.21.15.101:0/624443025 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f66b4102880 msgr2=0x7f66b40fbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:09.562 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.558+0000 7f66b37fe700 1 --2- 172.21.15.101:0/624443025 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f66b4102880 0x7f66b40fbc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:09.563 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.558+0000 7f66b37fe700 1 -- 172.21.15.101:0/624443025 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66b40ffd90 msgr2=0x7f66b40fb6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:09.563 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.558+0000 7f66b37fe700 1 --2- 172.21.15.101:0/624443025 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66b40ffd90 0x7f66b40fb6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:09.563 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.558+0000 7f66b37fe700 1 -- 172.21.15.101:0/624443025 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ac009940 con 0x7f66b4066cc0 2023-06-22T17:25:09.563 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.558+0000 7f66b37fe700 1 --2- 172.21.15.101:0/624443025 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f66b4066cc0 0x7f66b40fd060 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f66a800c890 tx=0x7f66a80043c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:25:09.563 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.558+0000 7f669bfff700 1 -- 172.21.15.101:0/624443025 <== mon.2 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a8012630 con 0x7f66b4066cc0 2023-06-22T17:25:09.563 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.558+0000 7f66b9f84700 1 -- 172.21.15.101:0/624443025 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66b411c070 con 0x7f66b4066cc0 2023-06-22T17:25:09.563 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.558+0000 7f669bfff700 1 -- 172.21.15.101:0/624443025 <== mon.2 v2:172.21.15.101:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f66a8012c70 con 0x7f66b4066cc0 2023-06-22T17:25:09.563 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.558+0000 7f669bfff700 1 -- 172.21.15.101:0/624443025 <== mon.2 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a8011c90 con 0x7f66b4066cc0 2023-06-22T17:25:09.564 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.558+0000 7f66b9f84700 1 -- 172.21.15.101:0/624443025 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b411c4a0 con 0x7f66b4066cc0 2023-06-22T17:25:09.564 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.559+0000 7f66b9f84700 1 -- 172.21.15.101:0/624443025 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66b40f8da0 con 0x7f66b4066cc0 2023-06-22T17:25:09.564 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.560+0000 7f669bfff700 1 -- 172.21.15.101:0/624443025 <== mon.2 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f66a8012790 con 0x7f66b4066cc0 2023-06-22T17:25:09.564 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.560+0000 7f669bfff700 1 --2- 172.21.15.101:0/624443025 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f669c09b680 0x7f669c09db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:25:09.565 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.560+0000 7f669bfff700 1 -- 172.21.15.101:0/624443025 <== mon.2 v2:172.21.15.101:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6257+0+0 (secure 0 0 0) 0x7f66a80c4720 con 0x7f66b4066cc0 2023-06-22T17:25:09.565 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.561+0000 7f66b2ffd700 1 --2- 172.21.15.101:0/624443025 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f669c09b680 0x7f669c09db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:25:09.565 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.561+0000 7f66b2ffd700 1 --2- 172.21.15.101:0/624443025 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f669c09b680 0x7f669c09db30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f66a4007fd0 tx=0x7f66a4009040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:25:09.565 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.565+0000 7f669bfff700 1 -- 172.21.15.101:0/624443025 <== mon.2 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f66a8090780 con 0x7f66b4066cc0 2023-06-22T17:25:09.758 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:09.757+0000 7f66b9f84700 1 -- 172.21.15.101:0/624443025 --> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f66b402b1c0 con 0x7f669c09b680 2023-06-22T17:25:10.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:09 smithi101 ceph-mon[85507]: purged_snaps scrub starts 2023-06-22T17:25:10.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:09 smithi101 ceph-mon[85507]: purged_snaps scrub ok 2023-06-22T17:25:10.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:09 smithi101 ceph-mon[85507]: osd.9 [v2:172.21.15.39:6826/3010986736,v1:172.21.15.39:6827/3010986736] boot 2023-06-22T17:25:10.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:09 smithi101 ceph-mon[85507]: osdmap e64: 10 total, 10 up, 10 in 2023-06-22T17:25:10.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:09 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-06-22T17:25:10.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:09 smithi101 ceph-mon[85507]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2023-06-22T17:25:10.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:09 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:25:10.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:09 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:25:10.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:09 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:25:10.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:09 smithi133 ceph-mon[86268]: purged_snaps scrub starts 2023-06-22T17:25:10.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:09 smithi133 ceph-mon[86268]: purged_snaps scrub ok 2023-06-22T17:25:10.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:09 smithi133 ceph-mon[86268]: osd.9 [v2:172.21.15.39:6826/3010986736,v1:172.21.15.39:6827/3010986736] boot 2023-06-22T17:25:10.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:09 smithi133 ceph-mon[86268]: osdmap e64: 10 total, 10 up, 10 in 2023-06-22T17:25:10.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:09 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-06-22T17:25:10.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:09 smithi133 ceph-mon[86268]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2023-06-22T17:25:10.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:09 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:25:10.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:09 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:25:10.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:09 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:25:10.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:09 smithi039 ceph-mon[84805]: purged_snaps scrub starts 2023-06-22T17:25:10.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:09 smithi039 ceph-mon[84805]: purged_snaps scrub ok 2023-06-22T17:25:10.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:09 smithi039 ceph-mon[84805]: osd.9 [v2:172.21.15.39:6826/3010986736,v1:172.21.15.39:6827/3010986736] boot 2023-06-22T17:25:10.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:09 smithi039 ceph-mon[84805]: osdmap e64: 10 total, 10 up, 10 in 2023-06-22T17:25:10.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:09 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-06-22T17:25:10.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:09 smithi039 ceph-mon[84805]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2023-06-22T17:25:10.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:09 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:25:10.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:09 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:25:10.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:09 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:25:11.256 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:10 smithi101 ceph-mon[85507]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:25:11.256 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:10 smithi101 ceph-mon[85507]: osdmap e65: 10 total, 10 up, 10 in 2023-06-22T17:25:11.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:10 smithi133 ceph-mon[86268]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:25:11.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:10 smithi133 ceph-mon[86268]: osdmap e65: 10 total, 10 up, 10 in 2023-06-22T17:25:11.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:10 smithi039 ceph-mon[84805]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:25:11.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:10 smithi039 ceph-mon[84805]: osdmap e65: 10 total, 10 up, 10 in 2023-06-22T17:25:12.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:12 smithi101 ceph-mon[85507]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-06-22T17:25:12.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:12 smithi101 ceph-mon[85507]: from='client.? 172.21.15.101:0/1011877259' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9da2943f-b947-49d6-bab1-bc3a95299996"}]: dispatch 2023-06-22T17:25:12.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:12 smithi101 ceph-mon[85507]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9da2943f-b947-49d6-bab1-bc3a95299996"}]: dispatch 2023-06-22T17:25:12.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:12 smithi101 ceph-mon[85507]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9da2943f-b947-49d6-bab1-bc3a95299996"}]': finished 2023-06-22T17:25:12.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:12 smithi101 ceph-mon[85507]: osdmap e66: 11 total, 10 up, 11 in 2023-06-22T17:25:12.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:12 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-06-22T17:25:12.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:12 smithi133 ceph-mon[86268]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-06-22T17:25:12.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:12 smithi133 ceph-mon[86268]: from='client.? 172.21.15.101:0/1011877259' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9da2943f-b947-49d6-bab1-bc3a95299996"}]: dispatch 2023-06-22T17:25:12.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:12 smithi133 ceph-mon[86268]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9da2943f-b947-49d6-bab1-bc3a95299996"}]: dispatch 2023-06-22T17:25:12.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:12 smithi133 ceph-mon[86268]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9da2943f-b947-49d6-bab1-bc3a95299996"}]': finished 2023-06-22T17:25:12.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:12 smithi133 ceph-mon[86268]: osdmap e66: 11 total, 10 up, 11 in 2023-06-22T17:25:12.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:12 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-06-22T17:25:12.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:12 smithi039 ceph-mon[84805]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-06-22T17:25:12.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:12 smithi039 ceph-mon[84805]: from='client.? 172.21.15.101:0/1011877259' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9da2943f-b947-49d6-bab1-bc3a95299996"}]: dispatch 2023-06-22T17:25:12.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:12 smithi039 ceph-mon[84805]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9da2943f-b947-49d6-bab1-bc3a95299996"}]: dispatch 2023-06-22T17:25:12.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:12 smithi039 ceph-mon[84805]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9da2943f-b947-49d6-bab1-bc3a95299996"}]': finished 2023-06-22T17:25:12.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:12 smithi039 ceph-mon[84805]: osdmap e66: 11 total, 10 up, 11 in 2023-06-22T17:25:12.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:12 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-06-22T17:25:13.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:13 smithi101 ceph-mon[85507]: from='client.? 172.21.15.101:0/382678575' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:25:13.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:13 smithi133 ceph-mon[86268]: from='client.? 172.21.15.101:0/382678575' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:25:13.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:13 smithi039 ceph-mon[84805]: from='client.? 172.21.15.101:0/382678575' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:25:14.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:14 smithi101 ceph-mon[85507]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-06-22T17:25:14.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:14 smithi133 ceph-mon[86268]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-06-22T17:25:14.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:14 smithi039 ceph-mon[84805]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-06-22T17:25:16.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:16 smithi101 ceph-mon[85507]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-06-22T17:25:16.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:16 smithi133 ceph-mon[86268]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-06-22T17:25:16.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:16 smithi039 ceph-mon[84805]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-06-22T17:25:18.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:18 smithi101 ceph-mon[85507]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-06-22T17:25:18.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:18 smithi133 ceph-mon[86268]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-06-22T17:25:18.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:18 smithi039 ceph-mon[84805]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-06-22T17:25:19.850 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:19 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-06-22T17:25:19.850 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:19 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:25:19.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:19 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-06-22T17:25:19.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:19 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:25:19.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:19 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-06-22T17:25:19.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:19 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:25:20.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:20 smithi101 ceph-mon[85507]: Deploying daemon osd.10 on smithi101 2023-06-22T17:25:20.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:20 smithi101 ceph-mon[85507]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-06-22T17:25:20.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:20 smithi133 ceph-mon[86268]: Deploying daemon osd.10 on smithi101 2023-06-22T17:25:20.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:20 smithi133 ceph-mon[86268]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-06-22T17:25:20.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:20 smithi039 ceph-mon[84805]: Deploying daemon osd.10 on smithi101 2023-06-22T17:25:20.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:20 smithi039 ceph-mon[84805]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-06-22T17:25:22.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:22 smithi101 ceph-mon[85507]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-06-22T17:25:22.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:22 smithi133 ceph-mon[86268]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-06-22T17:25:22.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:22 smithi039 ceph-mon[84805]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-06-22T17:25:24.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:24 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:24.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:24 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:24.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:24 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:25:24.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:24 smithi101 ceph-mon[85507]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-06-22T17:25:24.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:24 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:24.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:24 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:24.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:24 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:25:24.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:24 smithi133 ceph-mon[86268]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-06-22T17:25:24.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:24 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:24.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:24 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:24.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:24 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:25:24.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:24 smithi039 ceph-mon[84805]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-06-22T17:25:25.974 INFO:teuthology.orchestra.run.smithi101.stdout:Created osd(s) 10 on host 'smithi101' 2023-06-22T17:25:25.974 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:25.972+0000 7f669bfff700 1 -- 172.21.15.101:0/624443025 <== mgr.14150 v2:172.21.15.39:6800/886078282 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f66b402b1c0 con 0x7f669c09b680 2023-06-22T17:25:25.976 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:25.974+0000 7f66b9f84700 1 -- 172.21.15.101:0/624443025 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f669c09b680 msgr2=0x7f669c09db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:25.976 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:25.974+0000 7f66b9f84700 1 --2- 172.21.15.101:0/624443025 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f669c09b680 0x7f669c09db30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f66a4007fd0 tx=0x7f66a4009040 comp rx=0 tx=0).stop 2023-06-22T17:25:25.976 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:25.974+0000 7f66b9f84700 1 -- 172.21.15.101:0/624443025 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f66b4066cc0 msgr2=0x7f66b40fd060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:25.976 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:25.974+0000 7f66b9f84700 1 --2- 172.21.15.101:0/624443025 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f66b4066cc0 0x7f66b40fd060 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f66a800c890 tx=0x7f66a80043c0 comp rx=0 tx=0).stop 2023-06-22T17:25:25.977 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:25.974+0000 7f66b9f84700 1 -- 172.21.15.101:0/624443025 shutdown_connections 2023-06-22T17:25:25.977 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:25.974+0000 7f66b9f84700 1 --2- 172.21.15.101:0/624443025 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f669c09b680 0x7f669c09db30 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:25.977 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:25.974+0000 7f66b9f84700 1 --2- 172.21.15.101:0/624443025 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f66b4102880 0x7f66b40fbc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:25.977 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:25.974+0000 7f66b9f84700 1 --2- 172.21.15.101:0/624443025 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f66b4066cc0 0x7f66b40fd060 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:25.977 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:25.974+0000 7f66b9f84700 1 --2- 172.21.15.101:0/624443025 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66b40ffd90 0x7f66b40fb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:25.977 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:25.974+0000 7f66b9f84700 1 -- 172.21.15.101:0/624443025 >> 172.21.15.101:0/624443025 conn(0x7f66b40f7270 msgr2=0x7f66b40fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:25:25.977 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:25.974+0000 7f66b9f84700 1 -- 172.21.15.101:0/624443025 shutdown_connections 2023-06-22T17:25:25.978 INFO:teuthology.orchestra.run.smithi101.stderr:2023-06-22T17:25:25.974+0000 7f66b9f84700 1 -- 172.21.15.101:0/624443025 wait complete. 2023-06-22T17:25:26.446 DEBUG:teuthology.orchestra.run.smithi101:osd.10> sudo journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.10.service 2023-06-22T17:25:26.448 INFO:tasks.cephadm:Deploying osd.11 on smithi133 with /dev/vg_nvme/lv_1... 2023-06-22T17:25:26.448 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-06-22T17:25:26.475 INFO:journalctl@ceph.osd.10.smithi101.stdout:-- Logs begin at Thu 2023-06-22 17:06:34 UTC. -- 2023-06-22T17:25:26.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:26 smithi101 ceph-mon[85507]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-06-22T17:25:26.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:26 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:26.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:26 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:26.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:26 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:25:26.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:26 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:25:26.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:26 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:26.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:26 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:26.628 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:26 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:25:26.628 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:26 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:26.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:26 smithi133 ceph-mon[86268]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-06-22T17:25:26.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:26 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:26.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:26 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:26.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:26 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:25:26.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:26 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:25:26.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:26 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:26.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:26 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:26.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:26 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:25:26.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:26 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:26.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:26 smithi039 ceph-mon[84805]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-06-22T17:25:26.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:26 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:26.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:26 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:26.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:26 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:25:26.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:26 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:25:26.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:26 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:26.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:26 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:26.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:26 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:25:26.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:26 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:27.041 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.c/config 2023-06-22T17:25:27.399 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:25:27 smithi101 conmon[112983]: 2023-06-22T17:25:27.143+0000 7fe92e5fa540 -1 osd.10 0 log_to_monitors true 2023-06-22T17:25:27.588 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:27 smithi133 ceph-mon[86268]: from='osd.10 [v2:172.21.15.101:6824/3982067769,v1:172.21.15.101:6825/3982067769]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-06-22T17:25:27.588 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:27 smithi133 ceph-mon[86268]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-06-22T17:25:27.677 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:27 smithi101 ceph-mon[85507]: from='osd.10 [v2:172.21.15.101:6824/3982067769,v1:172.21.15.101:6825/3982067769]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-06-22T17:25:27.678 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:27 smithi101 ceph-mon[85507]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-06-22T17:25:27.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:27 smithi039 ceph-mon[84805]: from='osd.10 [v2:172.21.15.101:6824/3982067769,v1:172.21.15.101:6825/3982067769]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-06-22T17:25:27.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:27 smithi039 ceph-mon[84805]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-06-22T17:25:28.475 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:28 smithi101 ceph-mon[85507]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-06-22T17:25:28.476 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:28 smithi101 ceph-mon[85507]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-06-22T17:25:28.476 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:28 smithi101 ceph-mon[85507]: osdmap e67: 11 total, 10 up, 11 in 2023-06-22T17:25:28.476 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:28 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-06-22T17:25:28.476 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:28 smithi101 ceph-mon[85507]: from='osd.10 [v2:172.21.15.101:6824/3982067769,v1:172.21.15.101:6825/3982067769]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-06-22T17:25:28.476 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:28 smithi101 ceph-mon[85507]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-06-22T17:25:28.476 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:25:28 smithi101 conmon[112983]: 2023-06-22T17:25:28.457+0000 7fe924ddf700 -1 osd.10 0 waiting for initial osdmap 2023-06-22T17:25:28.533 INFO:teuthology.orchestra.run.smithi133.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-06-22T17:25:28.534 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-06-22T17:25:28.534 INFO:teuthology.orchestra.run.smithi133.stderr: stderr: 10+0 records in 2023-06-22T17:25:28.534 INFO:teuthology.orchestra.run.smithi133.stderr:10+0 records out 2023-06-22T17:25:28.534 INFO:teuthology.orchestra.run.smithi133.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112892 s, 929 MB/s 2023-06-22T17:25:28.534 INFO:teuthology.orchestra.run.smithi133.stderr:--> Zapping successful for: 2023-06-22T17:25:28.746 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:28 smithi133 ceph-mon[86268]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-06-22T17:25:28.747 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:28 smithi133 ceph-mon[86268]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-06-22T17:25:28.747 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:28 smithi133 ceph-mon[86268]: osdmap e67: 11 total, 10 up, 11 in 2023-06-22T17:25:28.747 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:28 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-06-22T17:25:28.747 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:28 smithi133 ceph-mon[86268]: from='osd.10 [v2:172.21.15.101:6824/3982067769,v1:172.21.15.101:6825/3982067769]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-06-22T17:25:28.747 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:28 smithi133 ceph-mon[86268]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-06-22T17:25:28.876 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:25:28 smithi101 conmon[112983]: 2023-06-22T17:25:28.473+0000 7fe91f3d9700 -1 osd.10 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-22T17:25:28.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:28 smithi039 ceph-mon[84805]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-06-22T17:25:28.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:28 smithi039 ceph-mon[84805]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-06-22T17:25:28.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:28 smithi039 ceph-mon[84805]: osdmap e67: 11 total, 10 up, 11 in 2023-06-22T17:25:28.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:28 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-06-22T17:25:28.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:28 smithi039 ceph-mon[84805]: from='osd.10 [v2:172.21.15.101:6824/3982067769,v1:172.21.15.101:6825/3982067769]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-06-22T17:25:28.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:28 smithi039 ceph-mon[84805]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-06-22T17:25:28.982 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph orch daemon add osd smithi133:vg_nvme/lv_1 2023-06-22T17:25:29.566 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.c/config 2023-06-22T17:25:29.772 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:29 smithi133 ceph-mon[86268]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2023-06-22T17:25:29.773 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:29 smithi133 ceph-mon[86268]: osdmap e68: 11 total, 10 up, 11 in 2023-06-22T17:25:29.773 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:29 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-06-22T17:25:29.773 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:29 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-06-22T17:25:29.773 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:29 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:29.773 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:29 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:29.774 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:29 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:29.774 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:29 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:29.774 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:29 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:29.774 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:29 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:29.774 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:29 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:29.774 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:29 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:25:29.774 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:29 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:25:29.774 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:29 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:29.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:29 smithi101 ceph-mon[85507]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2023-06-22T17:25:29.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:29 smithi101 ceph-mon[85507]: osdmap e68: 11 total, 10 up, 11 in 2023-06-22T17:25:29.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:29 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-06-22T17:25:29.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:29 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-06-22T17:25:29.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:29 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:29.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:29 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:29.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:29 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:29.878 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:29 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:29.878 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:29 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:29.878 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:29 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:29.878 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:29 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:29.878 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:29 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:25:29.878 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:29 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:25:29.878 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:29 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:29.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:29 smithi039 ceph-mon[84805]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2023-06-22T17:25:29.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:29 smithi039 ceph-mon[84805]: osdmap e68: 11 total, 10 up, 11 in 2023-06-22T17:25:29.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:29 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-06-22T17:25:29.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:29 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-06-22T17:25:29.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:29 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:29.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:29 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:29.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:29 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:29.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:29 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:29.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:29 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:29.935 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:29 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:29.935 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:29 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:29.935 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:29 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:25:29.935 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:29 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:25:29.935 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:29 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:30.868 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.865+0000 7f1db20f4700 1 -- 172.21.15.133:0/3709235145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1dac0fb450 msgr2=0x7f1dac105290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:30.901 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.865+0000 7f1db20f4700 1 --2- 172.21.15.133:0/3709235145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1dac0fb450 0x7f1dac105290 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1da4009d60 tx=0x7f1da4004d10 comp rx=0 tx=0).stop 2023-06-22T17:25:30.901 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.865+0000 7f1db20f4700 1 -- 172.21.15.133:0/3709235145 shutdown_connections 2023-06-22T17:25:30.901 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.865+0000 7f1db20f4700 1 --2- 172.21.15.133:0/3709235145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1dac0fb450 0x7f1dac105290 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:30.901 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.865+0000 7f1db20f4700 1 --2- 172.21.15.133:0/3709235145 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f1dac0fe290 0x7f1dac0fe6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:30.901 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.865+0000 7f1db20f4700 1 --2- 172.21.15.133:0/3709235145 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1dac0febe0 0x7f1dac0fad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:30.902 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.865+0000 7f1db20f4700 1 -- 172.21.15.133:0/3709235145 >> 172.21.15.133:0/3709235145 conn(0x7f1dac0f6a50 msgr2=0x7f1dac0f8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:25:30.902 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.865+0000 7f1db20f4700 1 -- 172.21.15.133:0/3709235145 shutdown_connections 2023-06-22T17:25:30.902 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.865+0000 7f1db20f4700 1 -- 172.21.15.133:0/3709235145 wait complete. 2023-06-22T17:25:30.902 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.865+0000 7f1db20f4700 1 Processor -- start 2023-06-22T17:25:30.902 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.865+0000 7f1db20f4700 1 -- start start 2023-06-22T17:25:30.902 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.866+0000 7f1db20f4700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f1dac0fb450 0x7f1dac0fd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:25:30.902 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.866+0000 7f1db20f4700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1dac0fe290 0x7f1dac0fbf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:25:30.902 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.866+0000 7f1db20f4700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1dac0febe0 0x7f1dac0fc490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:25:30.903 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.866+0000 7f1db20f4700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac0fcc00 con 0x7f1dac0febe0 2023-06-22T17:25:30.903 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.866+0000 7f1db20f4700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac0fcd70 con 0x7f1dac0fb450 2023-06-22T17:25:30.903 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.866+0000 7f1db20f4700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac117960 con 0x7f1dac0fe290 2023-06-22T17:25:30.903 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.866+0000 7f1da2ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1dac0fe290 0x7f1dac0fbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:25:30.903 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.866+0000 7f1da2ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1dac0fe290 0x7f1dac0fbf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:54110/0 (socket says 172.21.15.133:54110) 2023-06-22T17:25:30.903 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.866+0000 7f1da2ffd700 1 -- 172.21.15.133:0/2794378444 learned_addr learned my addr 172.21.15.133:0/2794378444 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-06-22T17:25:30.903 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.866+0000 7f1dab7fe700 1 --2- 172.21.15.133:0/2794378444 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f1dac0fb450 0x7f1dac0fd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:25:30.904 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.866+0000 7f1dabfff700 1 --2- 172.21.15.133:0/2794378444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1dac0febe0 0x7f1dac0fc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:25:30.904 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.866+0000 7f1da2ffd700 1 -- 172.21.15.133:0/2794378444 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f1dac0fb450 msgr2=0x7f1dac0fd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:30.904 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.866+0000 7f1da2ffd700 1 --2- 172.21.15.133:0/2794378444 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f1dac0fb450 0x7f1dac0fd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:30.904 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.866+0000 7f1da2ffd700 1 -- 172.21.15.133:0/2794378444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1dac0febe0 msgr2=0x7f1dac0fc490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:30.904 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.866+0000 7f1da2ffd700 1 --2- 172.21.15.133:0/2794378444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1dac0febe0 0x7f1dac0fc490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:30.904 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.866+0000 7f1da2ffd700 1 -- 172.21.15.133:0/2794378444 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da4009940 con 0x7f1dac0fe290 2023-06-22T17:25:30.904 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.866+0000 7f1da2ffd700 1 --2- 172.21.15.133:0/2794378444 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1dac0fe290 0x7f1dac0fbf50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1d9400b810 tx=0x7f1d94009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:25:30.904 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.866+0000 7f1da8ff9700 1 -- 172.21.15.133:0/2794378444 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9400f510 con 0x7f1dac0fe290 2023-06-22T17:25:30.905 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.866+0000 7f1db20f4700 1 -- 172.21.15.133:0/2794378444 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dac117c10 con 0x7f1dac0fe290 2023-06-22T17:25:30.905 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.866+0000 7f1da8ff9700 1 -- 172.21.15.133:0/2794378444 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f1d9400fb50 con 0x7f1dac0fe290 2023-06-22T17:25:30.905 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.866+0000 7f1da8ff9700 1 -- 172.21.15.133:0/2794378444 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d94011cd0 con 0x7f1dac0fe290 2023-06-22T17:25:30.905 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.867+0000 7f1db20f4700 1 -- 172.21.15.133:0/2794378444 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dac118150 con 0x7f1dac0fe290 2023-06-22T17:25:30.906 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:30 smithi101 ceph-mon[85507]: purged_snaps scrub starts 2023-06-22T17:25:30.906 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:30 smithi101 ceph-mon[85507]: purged_snaps scrub ok 2023-06-22T17:25:30.906 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:30 smithi101 ceph-mon[85507]: Detected new or changed devices on smithi101 2023-06-22T17:25:30.906 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:30 smithi101 ceph-mon[85507]: Adjusting osd_memory_target on smithi101 to 4292M 2023-06-22T17:25:30.906 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:30 smithi101 ceph-mon[85507]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-06-22T17:25:30.906 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:30 smithi101 ceph-mon[85507]: osd.10 [v2:172.21.15.101:6824/3982067769,v1:172.21.15.101:6825/3982067769] boot 2023-06-22T17:25:30.907 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:30 smithi101 ceph-mon[85507]: osdmap e69: 11 total, 11 up, 11 in 2023-06-22T17:25:30.907 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:30 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-06-22T17:25:30.907 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:30 smithi101 ceph-mon[85507]: osdmap e70: 11 total, 11 up, 11 in 2023-06-22T17:25:30.908 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.868+0000 7f1db20f4700 1 -- 172.21.15.133:0/2794378444 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dac1040f0 con 0x7f1dac0fe290 2023-06-22T17:25:30.908 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.868+0000 7f1da8ff9700 1 -- 172.21.15.133:0/2794378444 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f1d9400fcc0 con 0x7f1dac0fe290 2023-06-22T17:25:30.908 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.869+0000 7f1da8ff9700 1 --2- 172.21.15.133:0/2794378444 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f1d9809b680 0x7f1d9809db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:25:30.908 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.869+0000 7f1da8ff9700 1 -- 172.21.15.133:0/2794378444 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f1d940c43f0 con 0x7f1dac0fe290 2023-06-22T17:25:30.908 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.869+0000 7f1dab7fe700 1 --2- 172.21.15.133:0/2794378444 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f1d9809b680 0x7f1d9809db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:25:30.908 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.870+0000 7f1dab7fe700 1 --2- 172.21.15.133:0/2794378444 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f1d9809b680 0x7f1d9809db30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c007fd0 tx=0x7f1d9c009040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:25:30.909 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:30.873+0000 7f1da8ff9700 1 -- 172.21.15.133:0/2794378444 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f1d94090290 con 0x7f1dac0fe290 2023-06-22T17:25:30.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:30 smithi133 ceph-mon[86268]: purged_snaps scrub starts 2023-06-22T17:25:30.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:30 smithi133 ceph-mon[86268]: purged_snaps scrub ok 2023-06-22T17:25:30.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:30 smithi133 ceph-mon[86268]: Detected new or changed devices on smithi101 2023-06-22T17:25:30.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:30 smithi133 ceph-mon[86268]: Adjusting osd_memory_target on smithi101 to 4292M 2023-06-22T17:25:30.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:30 smithi133 ceph-mon[86268]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-06-22T17:25:30.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:30 smithi133 ceph-mon[86268]: osd.10 [v2:172.21.15.101:6824/3982067769,v1:172.21.15.101:6825/3982067769] boot 2023-06-22T17:25:30.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:30 smithi133 ceph-mon[86268]: osdmap e69: 11 total, 11 up, 11 in 2023-06-22T17:25:30.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:30 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-06-22T17:25:30.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:30 smithi133 ceph-mon[86268]: osdmap e70: 11 total, 11 up, 11 in 2023-06-22T17:25:30.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:30 smithi039 ceph-mon[84805]: purged_snaps scrub starts 2023-06-22T17:25:30.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:30 smithi039 ceph-mon[84805]: purged_snaps scrub ok 2023-06-22T17:25:30.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:30 smithi039 ceph-mon[84805]: Detected new or changed devices on smithi101 2023-06-22T17:25:30.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:30 smithi039 ceph-mon[84805]: Adjusting osd_memory_target on smithi101 to 4292M 2023-06-22T17:25:30.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:30 smithi039 ceph-mon[84805]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-06-22T17:25:30.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:30 smithi039 ceph-mon[84805]: osd.10 [v2:172.21.15.101:6824/3982067769,v1:172.21.15.101:6825/3982067769] boot 2023-06-22T17:25:30.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:30 smithi039 ceph-mon[84805]: osdmap e69: 11 total, 11 up, 11 in 2023-06-22T17:25:30.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:30 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-06-22T17:25:30.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:30 smithi039 ceph-mon[84805]: osdmap e70: 11 total, 11 up, 11 in 2023-06-22T17:25:31.064 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:31.063+0000 7f1db20f4700 1 -- 172.21.15.133:0/2794378444 --> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f1dac02b2c0 con 0x7f1d9809b680 2023-06-22T17:25:31.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:31 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:25:31.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:31 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:25:31.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:31 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:25:31.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:31 smithi101 ceph-mon[85507]: osdmap e71: 11 total, 11 up, 11 in 2023-06-22T17:25:31.887 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:31 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:25:31.887 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:31 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:25:31.887 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:31 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:25:31.887 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:31 smithi133 ceph-mon[86268]: osdmap e71: 11 total, 11 up, 11 in 2023-06-22T17:25:31.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:31 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-22T17:25:31.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:31 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-22T17:25:31.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:31 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:25:31.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:31 smithi039 ceph-mon[84805]: osdmap e71: 11 total, 11 up, 11 in 2023-06-22T17:25:32.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:32 smithi133 ceph-mon[86268]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:25:32.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:32 smithi133 ceph-mon[86268]: pgmap v212: 1 pgs: 1 remapped+peering; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-06-22T17:25:32.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:32 smithi101 ceph-mon[85507]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:25:32.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:32 smithi101 ceph-mon[85507]: pgmap v212: 1 pgs: 1 remapped+peering; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-06-22T17:25:32.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:32 smithi039 ceph-mon[84805]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:25:32.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:32 smithi039 ceph-mon[84805]: pgmap v212: 1 pgs: 1 remapped+peering; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-06-22T17:25:33.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:33 smithi101 ceph-mon[85507]: osdmap e72: 11 total, 11 up, 11 in 2023-06-22T17:25:33.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:33 smithi101 ceph-mon[85507]: from='client.? 172.21.15.133:0/3408480875' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee5d51b4-c9ad-4603-a967-c60f3c1f2cdc"}]: dispatch 2023-06-22T17:25:33.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:33 smithi101 ceph-mon[85507]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee5d51b4-c9ad-4603-a967-c60f3c1f2cdc"}]: dispatch 2023-06-22T17:25:33.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:33 smithi101 ceph-mon[85507]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ee5d51b4-c9ad-4603-a967-c60f3c1f2cdc"}]': finished 2023-06-22T17:25:33.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:33 smithi101 ceph-mon[85507]: osdmap e73: 12 total, 11 up, 12 in 2023-06-22T17:25:33.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:33 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-06-22T17:25:33.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:33 smithi133 ceph-mon[86268]: osdmap e72: 11 total, 11 up, 11 in 2023-06-22T17:25:33.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:33 smithi133 ceph-mon[86268]: from='client.? 172.21.15.133:0/3408480875' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee5d51b4-c9ad-4603-a967-c60f3c1f2cdc"}]: dispatch 2023-06-22T17:25:33.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:33 smithi133 ceph-mon[86268]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee5d51b4-c9ad-4603-a967-c60f3c1f2cdc"}]: dispatch 2023-06-22T17:25:33.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:33 smithi133 ceph-mon[86268]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ee5d51b4-c9ad-4603-a967-c60f3c1f2cdc"}]': finished 2023-06-22T17:25:33.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:33 smithi133 ceph-mon[86268]: osdmap e73: 12 total, 11 up, 12 in 2023-06-22T17:25:33.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:33 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-06-22T17:25:33.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:33 smithi039 ceph-mon[84805]: osdmap e72: 11 total, 11 up, 11 in 2023-06-22T17:25:33.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:33 smithi039 ceph-mon[84805]: from='client.? 172.21.15.133:0/3408480875' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee5d51b4-c9ad-4603-a967-c60f3c1f2cdc"}]: dispatch 2023-06-22T17:25:33.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:33 smithi039 ceph-mon[84805]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee5d51b4-c9ad-4603-a967-c60f3c1f2cdc"}]: dispatch 2023-06-22T17:25:33.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:33 smithi039 ceph-mon[84805]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ee5d51b4-c9ad-4603-a967-c60f3c1f2cdc"}]': finished 2023-06-22T17:25:33.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:33 smithi039 ceph-mon[84805]: osdmap e73: 12 total, 11 up, 12 in 2023-06-22T17:25:33.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:33 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-06-22T17:25:34.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:34 smithi101 ceph-mon[85507]: pgmap v215: 1 pgs: 1 remapped+peering; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-06-22T17:25:34.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:34 smithi101 ceph-mon[85507]: from='client.? 172.21.15.133:0/2504168650' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:25:34.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:34 smithi133 ceph-mon[86268]: pgmap v215: 1 pgs: 1 remapped+peering; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-06-22T17:25:34.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:34 smithi133 ceph-mon[86268]: from='client.? 172.21.15.133:0/2504168650' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:25:34.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:34 smithi039 ceph-mon[84805]: pgmap v215: 1 pgs: 1 remapped+peering; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-06-22T17:25:34.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:34 smithi039 ceph-mon[84805]: from='client.? 172.21.15.133:0/2504168650' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-22T17:25:36.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:36 smithi101 ceph-mon[85507]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 118 KiB/s, 0 objects/s recovering 2023-06-22T17:25:36.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:36 smithi133 ceph-mon[86268]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 118 KiB/s, 0 objects/s recovering 2023-06-22T17:25:36.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:36 smithi039 ceph-mon[84805]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 118 KiB/s, 0 objects/s recovering 2023-06-22T17:25:38.786 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:38 smithi133 ceph-mon[86268]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-06-22T17:25:38.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:38 smithi101 ceph-mon[85507]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-06-22T17:25:38.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:38 smithi039 ceph-mon[84805]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-06-22T17:25:40.776 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:40 smithi133 ceph-mon[86268]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 73 KiB/s, 0 objects/s recovering 2023-06-22T17:25:40.777 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:40 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-06-22T17:25:40.777 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:40 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:25:40.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:40 smithi101 ceph-mon[85507]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 73 KiB/s, 0 objects/s recovering 2023-06-22T17:25:40.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:40 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-06-22T17:25:40.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:40 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:25:40.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:40 smithi039 ceph-mon[84805]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 73 KiB/s, 0 objects/s recovering 2023-06-22T17:25:40.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:40 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-06-22T17:25:40.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:40 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:25:41.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:41 smithi101 ceph-mon[85507]: Deploying daemon osd.11 on smithi133 2023-06-22T17:25:41.897 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:41 smithi133 ceph-mon[86268]: Deploying daemon osd.11 on smithi133 2023-06-22T17:25:41.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:41 smithi039 ceph-mon[84805]: Deploying daemon osd.11 on smithi133 2023-06-22T17:25:42.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:42 smithi101 ceph-mon[85507]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 65 KiB/s, 0 objects/s recovering 2023-06-22T17:25:42.904 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:42 smithi133 ceph-mon[86268]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 65 KiB/s, 0 objects/s recovering 2023-06-22T17:25:42.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:42 smithi039 ceph-mon[84805]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 65 KiB/s, 0 objects/s recovering 2023-06-22T17:25:43.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:43 smithi133 ceph-mon[86268]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-06-22T17:25:43.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:43 smithi101 ceph-mon[85507]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-06-22T17:25:43.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:43 smithi039 ceph-mon[84805]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-06-22T17:25:44.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:44 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:25:44.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:44 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:44.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:44 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:44.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:44 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:25:44.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:44 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:44.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:44 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:44.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:44 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:25:44.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:44 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:44.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:44 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:45.791 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:45 smithi133 ceph-mon[86268]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-06-22T17:25:45.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:45 smithi101 ceph-mon[85507]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-06-22T17:25:45.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:45 smithi039 ceph-mon[84805]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-06-22T17:25:46.976 INFO:teuthology.orchestra.run.smithi133.stdout:Created osd(s) 11 on host 'smithi133' 2023-06-22T17:25:46.977 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:46.974+0000 7f1da8ff9700 1 -- 172.21.15.133:0/2794378444 <== mgr.14150 v2:172.21.15.39:6800/886078282 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f1dac02b2c0 con 0x7f1d9809b680 2023-06-22T17:25:46.978 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:46.976+0000 7f1db20f4700 1 -- 172.21.15.133:0/2794378444 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f1d9809b680 msgr2=0x7f1d9809db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:46.978 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:46.976+0000 7f1db20f4700 1 --2- 172.21.15.133:0/2794378444 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f1d9809b680 0x7f1d9809db30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c007fd0 tx=0x7f1d9c009040 comp rx=0 tx=0).stop 2023-06-22T17:25:46.978 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:46.976+0000 7f1db20f4700 1 -- 172.21.15.133:0/2794378444 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1dac0fe290 msgr2=0x7f1dac0fbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:46.979 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:46.976+0000 7f1db20f4700 1 --2- 172.21.15.133:0/2794378444 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1dac0fe290 0x7f1dac0fbf50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1d9400b810 tx=0x7f1d94009250 comp rx=0 tx=0).stop 2023-06-22T17:25:46.979 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:46.976+0000 7f1db20f4700 1 -- 172.21.15.133:0/2794378444 shutdown_connections 2023-06-22T17:25:46.979 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:46.976+0000 7f1db20f4700 1 --2- 172.21.15.133:0/2794378444 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f1d9809b680 0x7f1d9809db30 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:46.979 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:46.976+0000 7f1db20f4700 1 --2- 172.21.15.133:0/2794378444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1dac0febe0 0x7f1dac0fc490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:46.979 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:46.976+0000 7f1db20f4700 1 --2- 172.21.15.133:0/2794378444 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f1dac0fb450 0x7f1dac0fd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:46.979 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:46.976+0000 7f1db20f4700 1 --2- 172.21.15.133:0/2794378444 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1dac0fe290 0x7f1dac0fbf50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:46.979 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:46.976+0000 7f1db20f4700 1 -- 172.21.15.133:0/2794378444 >> 172.21.15.133:0/2794378444 conn(0x7f1dac0f6a50 msgr2=0x7f1dac101270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:25:46.980 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:46.976+0000 7f1db20f4700 1 -- 172.21.15.133:0/2794378444 shutdown_connections 2023-06-22T17:25:46.980 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:46.976+0000 7f1db20f4700 1 -- 172.21.15.133:0/2794378444 wait complete. 2023-06-22T17:25:47.314 DEBUG:teuthology.orchestra.run.smithi133:osd.11> sudo journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.11.service 2023-06-22T17:25:47.317 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2023-06-22T17:25:47.317 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph osd stat -f json 2023-06-22T17:25:47.345 INFO:journalctl@ceph.osd.11.smithi133.stdout:-- Logs begin at Thu 2023-06-22 17:06:34 UTC. -- 2023-06-22T17:25:47.927 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-22T17:25:48.168 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:47 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:48.168 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:47 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:48.169 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:47 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:48.169 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:47 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:48.169 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:47 smithi133 ceph-mon[86268]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2023-06-22T17:25:48.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:47 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:48.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:47 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:48.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:47 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:48.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:47 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:48.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:47 smithi039 ceph-mon[84805]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2023-06-22T17:25:48.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:47 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:48.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:47 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:48.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:47 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:48.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:47 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:48.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:47 smithi101 ceph-mon[85507]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2023-06-22T17:25:48.420 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:25:48 smithi133 conmon[115049]: 2023-06-22T17:25:48.218+0000 7f145a186540 -1 osd.11 0 log_to_monitors true 2023-06-22T17:25:49.213 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.208+0000 7f36172be700 1 -- 172.21.15.39:0/354654660 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36100fbb10 msgr2=0x7f36100fbf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:49.213 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.208+0000 7f36172be700 1 --2- 172.21.15.39:0/354654660 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36100fbb10 0x7f36100fbf20 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f360c0049c0 tx=0x7f360c009e30 comp rx=0 tx=0).stop 2023-06-22T17:25:49.213 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.208+0000 7f36172be700 1 -- 172.21.15.39:0/354654660 shutdown_connections 2023-06-22T17:25:49.214 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.208+0000 7f36172be700 1 --2- 172.21.15.39:0/354654660 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f36100fd6b0 0x7f3610103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:49.214 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.208+0000 7f36172be700 1 --2- 172.21.15.39:0/354654660 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36100fbb10 0x7f36100fbf20 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:49.214 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.208+0000 7f36172be700 1 --2- 172.21.15.39:0/354654660 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f36100fcd20 0x7f36100fd170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:49.214 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.208+0000 7f36172be700 1 -- 172.21.15.39:0/354654660 >> 172.21.15.39:0/354654660 conn(0x7f36100f7270 msgr2=0x7f36100f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:25:49.215 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.208+0000 7f36172be700 1 -- 172.21.15.39:0/354654660 shutdown_connections 2023-06-22T17:25:49.215 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.208+0000 7f36172be700 1 -- 172.21.15.39:0/354654660 wait complete. 2023-06-22T17:25:49.215 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.208+0000 7f36172be700 1 Processor -- start 2023-06-22T17:25:49.215 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.208+0000 7f36172be700 1 -- start start 2023-06-22T17:25:49.215 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.208+0000 7f36172be700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f36100fbb10 0x7f361010d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:25:49.216 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.209+0000 7f36172be700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f36100fcd20 0x7f361010de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:25:49.216 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.209+0000 7f36172be700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36100fd6b0 0x7f3610114f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:25:49.216 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.209+0000 7f36172be700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3610115520 con 0x7f36100fd6b0 2023-06-22T17:25:49.217 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.209+0000 7f36172be700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3610115690 con 0x7f36100fcd20 2023-06-22T17:25:49.217 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.209+0000 7f36172be700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3610115850 con 0x7f36100fbb10 2023-06-22T17:25:49.217 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.209+0000 7f361585b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36100fd6b0 0x7f3610114f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:25:49.217 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.209+0000 7f361585b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36100fd6b0 0x7f3610114f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34426/0 (socket says 172.21.15.39:34426) 2023-06-22T17:25:49.217 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.209+0000 7f361585b700 1 -- 172.21.15.39:0/222560989 learned_addr learned my addr 172.21.15.39:0/222560989 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:25:49.218 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.209+0000 7f361505a700 1 --2- 172.21.15.39:0/222560989 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f36100fbb10 0x7f361010d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:25:49.218 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.209+0000 7f3614859700 1 --2- 172.21.15.39:0/222560989 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f36100fcd20 0x7f361010de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:25:49.218 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.209+0000 7f361585b700 1 -- 172.21.15.39:0/222560989 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f36100fbb10 msgr2=0x7f361010d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:49.218 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.209+0000 7f361585b700 1 --2- 172.21.15.39:0/222560989 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f36100fbb10 0x7f361010d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:49.218 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.209+0000 7f361585b700 1 -- 172.21.15.39:0/222560989 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f36100fcd20 msgr2=0x7f361010de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:49.218 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.209+0000 7f361585b700 1 --2- 172.21.15.39:0/222560989 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f36100fcd20 0x7f361010de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:49.218 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.209+0000 7f361585b700 1 -- 172.21.15.39:0/222560989 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f360c005040 con 0x7f36100fd6b0 2023-06-22T17:25:49.218 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.209+0000 7f361585b700 1 --2- 172.21.15.39:0/222560989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36100fd6b0 0x7f3610114f10 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f360800ec30 tx=0x7f360800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:25:49.219 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.209+0000 7f3605ffb700 1 -- 172.21.15.39:0/222560989 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3608007b30 con 0x7f36100fd6b0 2023-06-22T17:25:49.219 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.209+0000 7f36172be700 1 -- 172.21.15.39:0/222560989 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3610115b30 con 0x7f36100fd6b0 2023-06-22T17:25:49.219 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.209+0000 7f3605ffb700 1 -- 172.21.15.39:0/222560989 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f360800f040 con 0x7f36100fd6b0 2023-06-22T17:25:49.219 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.209+0000 7f3605ffb700 1 -- 172.21.15.39:0/222560989 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360800caf0 con 0x7f36100fd6b0 2023-06-22T17:25:49.219 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.210+0000 7f36172be700 1 -- 172.21.15.39:0/222560989 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3610116070 con 0x7f36100fd6b0 2023-06-22T17:25:49.219 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.211+0000 7f36172be700 1 -- 172.21.15.39:0/222560989 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3610107440 con 0x7f36100fd6b0 2023-06-22T17:25:49.220 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.212+0000 7f3605ffb700 1 -- 172.21.15.39:0/222560989 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f360801d400 con 0x7f36100fd6b0 2023-06-22T17:25:49.220 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.212+0000 7f3605ffb700 1 --2- 172.21.15.39:0/222560989 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f35fc09b680 0x7f35fc09db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:25:49.220 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.213+0000 7f361505a700 1 --2- 172.21.15.39:0/222560989 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f35fc09b680 0x7f35fc09db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:25:49.221 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.213+0000 7f3605ffb700 1 -- 172.21.15.39:0/222560989 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6827+0+0 (secure 0 0 0) 0x7f36080c4bc0 con 0x7f36100fd6b0 2023-06-22T17:25:49.221 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.213+0000 7f361505a700 1 --2- 172.21.15.39:0/222560989 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f35fc09b680 0x7f35fc09db30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f360c009fd0 tx=0x7f360c009e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:25:49.221 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.217+0000 7f3605ffb700 1 -- 172.21.15.39:0/222560989 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f36080909f0 con 0x7f36100fd6b0 2023-06-22T17:25:49.293 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:49 smithi133 ceph-mon[86268]: from='osd.11 [v2:172.21.15.133:6824/1265049047,v1:172.21.15.133:6825/1265049047]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-06-22T17:25:49.293 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:49 smithi133 ceph-mon[86268]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-06-22T17:25:49.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:49 smithi101 ceph-mon[85507]: from='osd.11 [v2:172.21.15.133:6824/1265049047,v1:172.21.15.133:6825/1265049047]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-06-22T17:25:49.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:49 smithi101 ceph-mon[85507]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-06-22T17:25:49.411 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.409+0000 7f36172be700 1 -- 172.21.15.39:0/222560989 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f3610116260 con 0x7f36100fd6b0 2023-06-22T17:25:49.412 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.410+0000 7f3605ffb700 1 -- 172.21.15.39:0/222560989 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v74) v1 ==== 74+0+133 (secure 0 0 0) 0x7f3608094010 con 0x7f36100fd6b0 2023-06-22T17:25:49.412 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:25:49.414 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.412+0000 7f36172be700 1 -- 172.21.15.39:0/222560989 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f35fc09b680 msgr2=0x7f35fc09db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:49.414 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.412+0000 7f36172be700 1 --2- 172.21.15.39:0/222560989 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f35fc09b680 0x7f35fc09db30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f360c009fd0 tx=0x7f360c009e30 comp rx=0 tx=0).stop 2023-06-22T17:25:49.414 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.412+0000 7f36172be700 1 -- 172.21.15.39:0/222560989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36100fd6b0 msgr2=0x7f3610114f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:49.414 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.412+0000 7f36172be700 1 --2- 172.21.15.39:0/222560989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36100fd6b0 0x7f3610114f10 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f360800ec30 tx=0x7f360800b040 comp rx=0 tx=0).stop 2023-06-22T17:25:49.415 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.412+0000 7f36172be700 1 -- 172.21.15.39:0/222560989 shutdown_connections 2023-06-22T17:25:49.415 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.412+0000 7f36172be700 1 --2- 172.21.15.39:0/222560989 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f35fc09b680 0x7f35fc09db30 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:49.415 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.412+0000 7f36172be700 1 --2- 172.21.15.39:0/222560989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36100fd6b0 0x7f3610114f10 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:49.415 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.412+0000 7f36172be700 1 --2- 172.21.15.39:0/222560989 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f36100fbb10 0x7f361010d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:49.415 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.412+0000 7f36172be700 1 --2- 172.21.15.39:0/222560989 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f36100fcd20 0x7f361010de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:49.415 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.412+0000 7f36172be700 1 -- 172.21.15.39:0/222560989 >> 172.21.15.39:0/222560989 conn(0x7f36100f7270 msgr2=0x7f3610065fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:25:49.415 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.412+0000 7f36172be700 1 -- 172.21.15.39:0/222560989 shutdown_connections 2023-06-22T17:25:49.415 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:49.412+0000 7f36172be700 1 -- 172.21.15.39:0/222560989 wait complete. 2023-06-22T17:25:49.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:49 smithi039 ceph-mon[84805]: from='osd.11 [v2:172.21.15.133:6824/1265049047,v1:172.21.15.133:6825/1265049047]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-06-22T17:25:49.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:49 smithi039 ceph-mon[84805]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-06-22T17:25:49.866 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":74,"num_osds":12,"num_up_osds":11,"osd_up_since":1687454729,"num_in_osds":12,"osd_in_since":1687454733,"num_remapped_pgs":0} 2023-06-22T17:25:50.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:50 smithi039 ceph-mon[84805]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-06-22T17:25:50.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:50 smithi039 ceph-mon[84805]: osdmap e74: 12 total, 11 up, 12 in 2023-06-22T17:25:50.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:50 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-06-22T17:25:50.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:50 smithi039 ceph-mon[84805]: from='osd.11 [v2:172.21.15.133:6824/1265049047,v1:172.21.15.133:6825/1265049047]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2023-06-22T17:25:50.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:50 smithi039 ceph-mon[84805]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2023-06-22T17:25:50.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:50 smithi039 ceph-mon[84805]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2023-06-22T17:25:50.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:50 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/222560989' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-06-22T17:25:50.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:50 smithi039 ceph-mon[84805]: Detected new or changed devices on smithi133 2023-06-22T17:25:50.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:50 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:50.185 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:50 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:50.185 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:50 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:50.185 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:50 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:50.185 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:50 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:50.185 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:50 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:50.185 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:50 smithi039 ceph-mon[84805]: Adjusting osd_memory_target on smithi133 to 4292M 2023-06-22T17:25:50.185 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:50 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:50.186 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:50 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:25:50.186 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:50 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:25:50.186 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:50 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:50.186 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:50 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:25:50.186 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:50 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:25:50.186 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:50 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:25:50.186 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:50 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:50.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:50 smithi101 ceph-mon[85507]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-06-22T17:25:50.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:50 smithi101 ceph-mon[85507]: osdmap e74: 12 total, 11 up, 12 in 2023-06-22T17:25:50.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:50 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-06-22T17:25:50.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:50 smithi101 ceph-mon[85507]: from='osd.11 [v2:172.21.15.133:6824/1265049047,v1:172.21.15.133:6825/1265049047]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2023-06-22T17:25:50.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:50 smithi101 ceph-mon[85507]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2023-06-22T17:25:50.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:50 smithi101 ceph-mon[85507]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2023-06-22T17:25:50.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:50 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/222560989' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-06-22T17:25:50.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:50 smithi101 ceph-mon[85507]: Detected new or changed devices on smithi133 2023-06-22T17:25:50.378 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:50 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:50.378 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:50 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:50.378 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:50 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:50.378 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:50 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:50.378 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:50 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:50.378 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:50 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:50.378 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:50 smithi101 ceph-mon[85507]: Adjusting osd_memory_target on smithi133 to 4292M 2023-06-22T17:25:50.379 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:50 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:50.379 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:50 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:25:50.379 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:50 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:25:50.379 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:50 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:50.379 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:50 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:25:50.379 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:50 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:25:50.379 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:50 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:25:50.379 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:50 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:50.420 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:25:50 smithi133 conmon[115049]: 2023-06-22T17:25:50.030+0000 7f145096b700 -1 osd.11 0 waiting for initial osdmap 2023-06-22T17:25:50.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:25:50 smithi133 conmon[115049]: 2023-06-22T17:25:50.047+0000 7f144d76a700 -1 osd.11 75 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-22T17:25:50.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:50 smithi133 ceph-mon[86268]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-06-22T17:25:50.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:50 smithi133 ceph-mon[86268]: osdmap e74: 12 total, 11 up, 12 in 2023-06-22T17:25:50.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:50 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-06-22T17:25:50.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:50 smithi133 ceph-mon[86268]: from='osd.11 [v2:172.21.15.133:6824/1265049047,v1:172.21.15.133:6825/1265049047]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2023-06-22T17:25:50.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:50 smithi133 ceph-mon[86268]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2023-06-22T17:25:50.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:50 smithi133 ceph-mon[86268]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2023-06-22T17:25:50.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:50 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/222560989' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-06-22T17:25:50.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:50 smithi133 ceph-mon[86268]: Detected new or changed devices on smithi133 2023-06-22T17:25:50.422 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:50 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:50.423 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:50 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:50.423 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:50 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:50.423 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:50 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:50.423 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:50 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:50.423 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:50 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-06-22T17:25:50.423 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:50 smithi133 ceph-mon[86268]: Adjusting osd_memory_target on smithi133 to 4292M 2023-06-22T17:25:50.423 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:50 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:50.423 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:50 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:25:50.424 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:50 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:25:50.424 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:50 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:50.424 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:50 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:25:50.424 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:50 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:25:50.424 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:50 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:25:50.424 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:50 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:25:50.867 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph osd stat -f json 2023-06-22T17:25:51.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:51 smithi039 ceph-mon[84805]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]': finished 2023-06-22T17:25:51.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:51 smithi039 ceph-mon[84805]: osdmap e75: 12 total, 11 up, 12 in 2023-06-22T17:25:51.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:51 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-06-22T17:25:51.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:51 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-06-22T17:25:51.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:51 smithi101 ceph-mon[85507]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]': finished 2023-06-22T17:25:51.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:51 smithi101 ceph-mon[85507]: osdmap e75: 12 total, 11 up, 12 in 2023-06-22T17:25:51.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:51 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-06-22T17:25:51.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:51 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-06-22T17:25:51.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:51 smithi133 ceph-mon[86268]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]': finished 2023-06-22T17:25:51.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:51 smithi133 ceph-mon[86268]: osdmap e75: 12 total, 11 up, 12 in 2023-06-22T17:25:51.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:51 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-06-22T17:25:51.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:51 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-06-22T17:25:51.477 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-22T17:25:52.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:52 smithi101 ceph-mon[85507]: purged_snaps scrub starts 2023-06-22T17:25:52.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:52 smithi101 ceph-mon[85507]: purged_snaps scrub ok 2023-06-22T17:25:52.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:52 smithi101 ceph-mon[85507]: osd.11 [v2:172.21.15.133:6824/1265049047,v1:172.21.15.133:6825/1265049047] boot 2023-06-22T17:25:52.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:52 smithi101 ceph-mon[85507]: osdmap e76: 12 total, 12 up, 12 in 2023-06-22T17:25:52.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:52 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-06-22T17:25:52.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:52 smithi101 ceph-mon[85507]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:25:52.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:52 smithi133 ceph-mon[86268]: purged_snaps scrub starts 2023-06-22T17:25:52.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:52 smithi133 ceph-mon[86268]: purged_snaps scrub ok 2023-06-22T17:25:52.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:52 smithi133 ceph-mon[86268]: osd.11 [v2:172.21.15.133:6824/1265049047,v1:172.21.15.133:6825/1265049047] boot 2023-06-22T17:25:52.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:52 smithi133 ceph-mon[86268]: osdmap e76: 12 total, 12 up, 12 in 2023-06-22T17:25:52.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:52 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-06-22T17:25:52.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:52 smithi133 ceph-mon[86268]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:25:52.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:52 smithi039 ceph-mon[84805]: purged_snaps scrub starts 2023-06-22T17:25:52.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:52 smithi039 ceph-mon[84805]: purged_snaps scrub ok 2023-06-22T17:25:52.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:52 smithi039 ceph-mon[84805]: osd.11 [v2:172.21.15.133:6824/1265049047,v1:172.21.15.133:6825/1265049047] boot 2023-06-22T17:25:52.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:52 smithi039 ceph-mon[84805]: osdmap e76: 12 total, 12 up, 12 in 2023-06-22T17:25:52.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:52 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-06-22T17:25:52.434 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:52 smithi039 ceph-mon[84805]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:25:52.685 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.683+0000 7fe6ae53a700 1 -- 172.21.15.39:0/4656457 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6a80fbb00 msgr2=0x7fe6a80fbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:52.685 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.683+0000 7fe6ae53a700 1 --2- 172.21.15.39:0/4656457 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6a80fbb00 0x7fe6a80fbf10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe69c0049c0 tx=0x7fe69c009e30 comp rx=0 tx=0).stop 2023-06-22T17:25:52.686 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.683+0000 7fe6ae53a700 1 -- 172.21.15.39:0/4656457 shutdown_connections 2023-06-22T17:25:52.686 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.683+0000 7fe6ae53a700 1 --2- 172.21.15.39:0/4656457 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe6a80fd6a0 0x7fe6a8103f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:52.686 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.683+0000 7fe6ae53a700 1 --2- 172.21.15.39:0/4656457 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6a80fbb00 0x7fe6a80fbf10 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:52.686 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.683+0000 7fe6ae53a700 1 --2- 172.21.15.39:0/4656457 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe6a80fcd10 0x7fe6a80fd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:52.686 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.683+0000 7fe6ae53a700 1 -- 172.21.15.39:0/4656457 >> 172.21.15.39:0/4656457 conn(0x7fe6a80f7240 msgr2=0x7fe6a80f9690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:25:52.686 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.683+0000 7fe6ae53a700 1 -- 172.21.15.39:0/4656457 shutdown_connections 2023-06-22T17:25:52.686 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.683+0000 7fe6ae53a700 1 -- 172.21.15.39:0/4656457 wait complete. 2023-06-22T17:25:52.687 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.684+0000 7fe6ae53a700 1 Processor -- start 2023-06-22T17:25:52.688 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.684+0000 7fe6ae53a700 1 -- start start 2023-06-22T17:25:52.688 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.684+0000 7fe6ae53a700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe6a80fbb00 0x7fe6a8066ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:25:52.689 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.684+0000 7fe6ae53a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe6a80fcd10 0x7fe6a8112860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:25:52.689 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.684+0000 7fe6ae53a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6a80fd6a0 0x7fe6a8114e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:25:52.689 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.684+0000 7fe6ae53a700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a8115670 con 0x7fe6a80fd6a0 2023-06-22T17:25:52.689 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.684+0000 7fe6ae53a700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a81157e0 con 0x7fe6a80fcd10 2023-06-22T17:25:52.689 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.684+0000 7fe6ae53a700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a81159a0 con 0x7fe6a80fbb00 2023-06-22T17:25:52.689 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.684+0000 7fe6acad7700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6a80fd6a0 0x7fe6a8114e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:25:52.689 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.684+0000 7fe6acad7700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6a80fd6a0 0x7fe6a8114e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34446/0 (socket says 172.21.15.39:34446) 2023-06-22T17:25:52.689 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.684+0000 7fe6acad7700 1 -- 172.21.15.39:0/586858399 learned_addr learned my addr 172.21.15.39:0/586858399 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:25:52.690 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.684+0000 7fe6a77fe700 1 --2- 172.21.15.39:0/586858399 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe6a80fcd10 0x7fe6a8112860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:25:52.690 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.684+0000 7fe6a7fff700 1 --2- 172.21.15.39:0/586858399 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe6a80fbb00 0x7fe6a8066ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:25:52.690 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.684+0000 7fe6acad7700 1 -- 172.21.15.39:0/586858399 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe6a80fbb00 msgr2=0x7fe6a8066ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:52.690 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.684+0000 7fe6acad7700 1 --2- 172.21.15.39:0/586858399 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe6a80fbb00 0x7fe6a8066ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:52.690 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.685+0000 7fe6acad7700 1 -- 172.21.15.39:0/586858399 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe6a80fcd10 msgr2=0x7fe6a8112860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:52.690 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.685+0000 7fe6acad7700 1 --2- 172.21.15.39:0/586858399 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe6a80fcd10 0x7fe6a8112860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:52.690 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.685+0000 7fe6acad7700 1 -- 172.21.15.39:0/586858399 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe69c005040 con 0x7fe6a80fd6a0 2023-06-22T17:25:52.690 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.685+0000 7fe6acad7700 1 --2- 172.21.15.39:0/586858399 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6a80fd6a0 0x7fe6a8114e80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe6a000bc30 tx=0x7fe6a0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:25:52.691 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.685+0000 7fe6a4ff9700 1 -- 172.21.15.39:0/586858399 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a00092e0 con 0x7fe6a80fd6a0 2023-06-22T17:25:52.691 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.685+0000 7fe6a4ff9700 1 -- 172.21.15.39:0/586858399 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7fe6a000cb10 con 0x7fe6a80fd6a0 2023-06-22T17:25:52.691 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.685+0000 7fe6ae53a700 1 -- 172.21.15.39:0/586858399 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6a8115c20 con 0x7fe6a80fd6a0 2023-06-22T17:25:52.691 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.685+0000 7fe6a4ff9700 1 -- 172.21.15.39:0/586858399 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a0015c10 con 0x7fe6a80fd6a0 2023-06-22T17:25:52.691 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.685+0000 7fe6ae53a700 1 -- 172.21.15.39:0/586858399 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6a81160c0 con 0x7fe6a80fd6a0 2023-06-22T17:25:52.691 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.686+0000 7fe6ae53a700 1 -- 172.21.15.39:0/586858399 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6a81074b0 con 0x7fe6a80fd6a0 2023-06-22T17:25:52.692 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.687+0000 7fe6a4ff9700 1 -- 172.21.15.39:0/586858399 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7fe6a0012070 con 0x7fe6a80fd6a0 2023-06-22T17:25:52.692 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.688+0000 7fe6a4ff9700 1 --2- 172.21.15.39:0/586858399 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fe69009b680 0x7fe69009db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:25:52.692 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.688+0000 7fe6a7fff700 1 --2- 172.21.15.39:0/586858399 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fe69009b680 0x7fe69009db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:25:52.692 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.688+0000 7fe6a4ff9700 1 -- 172.21.15.39:0/586858399 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fe6a00c4560 con 0x7fe6a80fd6a0 2023-06-22T17:25:52.692 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.688+0000 7fe6a7fff700 1 --2- 172.21.15.39:0/586858399 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fe69009b680 0x7fe69009db30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe69c009f00 tx=0x7fe69c009e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:25:52.693 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.692+0000 7fe6a4ff9700 1 -- 172.21.15.39:0/586858399 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fe6a0090260 con 0x7fe6a80fd6a0 2023-06-22T17:25:52.886 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.884+0000 7fe6ae53a700 1 -- 172.21.15.39:0/586858399 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fe6a8102990 con 0x7fe6a80fd6a0 2023-06-22T17:25:52.887 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.885+0000 7fe6a4ff9700 1 -- 172.21.15.39:0/586858399 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v77) v1 ==== 74+0+133 (secure 0 0 0) 0x7fe6a000cc80 con 0x7fe6a80fd6a0 2023-06-22T17:25:52.887 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:25:52.889 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.887+0000 7fe6ae53a700 1 -- 172.21.15.39:0/586858399 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fe69009b680 msgr2=0x7fe69009db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:52.889 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.887+0000 7fe6ae53a700 1 --2- 172.21.15.39:0/586858399 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fe69009b680 0x7fe69009db30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe69c009f00 tx=0x7fe69c009e30 comp rx=0 tx=0).stop 2023-06-22T17:25:52.889 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.887+0000 7fe6ae53a700 1 -- 172.21.15.39:0/586858399 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6a80fd6a0 msgr2=0x7fe6a8114e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:52.889 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.887+0000 7fe6ae53a700 1 --2- 172.21.15.39:0/586858399 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6a80fd6a0 0x7fe6a8114e80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe6a000bc30 tx=0x7fe6a0009840 comp rx=0 tx=0).stop 2023-06-22T17:25:52.889 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.887+0000 7fe6ae53a700 1 -- 172.21.15.39:0/586858399 shutdown_connections 2023-06-22T17:25:52.890 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.887+0000 7fe6ae53a700 1 --2- 172.21.15.39:0/586858399 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fe69009b680 0x7fe69009db30 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:52.890 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.887+0000 7fe6ae53a700 1 --2- 172.21.15.39:0/586858399 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6a80fd6a0 0x7fe6a8114e80 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:52.890 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.887+0000 7fe6ae53a700 1 --2- 172.21.15.39:0/586858399 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe6a80fbb00 0x7fe6a8066ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:52.890 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.887+0000 7fe6ae53a700 1 --2- 172.21.15.39:0/586858399 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe6a80fcd10 0x7fe6a8112860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:52.890 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.887+0000 7fe6ae53a700 1 -- 172.21.15.39:0/586858399 >> 172.21.15.39:0/586858399 conn(0x7fe6a80f7240 msgr2=0x7fe6a80f94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:25:52.890 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.887+0000 7fe6ae53a700 1 -- 172.21.15.39:0/586858399 shutdown_connections 2023-06-22T17:25:52.890 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:52.887+0000 7fe6ae53a700 1 -- 172.21.15.39:0/586858399 wait complete. 2023-06-22T17:25:53.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:53 smithi039 ceph-mon[84805]: osdmap e77: 12 total, 12 up, 12 in 2023-06-22T17:25:53.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:53 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/586858399' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-06-22T17:25:53.328 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":77,"num_osds":12,"num_up_osds":12,"osd_up_since":1687454751,"num_in_osds":12,"osd_in_since":1687454733,"num_remapped_pgs":0} 2023-06-22T17:25:53.328 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph osd dump --format=json 2023-06-22T17:25:53.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:53 smithi101 ceph-mon[85507]: osdmap e77: 12 total, 12 up, 12 in 2023-06-22T17:25:53.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:53 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/586858399' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-06-22T17:25:53.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:53 smithi133 ceph-mon[86268]: osdmap e77: 12 total, 12 up, 12 in 2023-06-22T17:25:53.421 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:53 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/586858399' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-06-22T17:25:53.921 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-22T17:25:54.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:54 smithi101 ceph-mon[85507]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:25:54.420 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:54 smithi133 ceph-mon[86268]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:25:54.433 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:54 smithi039 ceph-mon[84805]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:25:55.161 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.158+0000 7fedce30b700 1 -- 172.21.15.39:0/2594936778 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedc80fbb30 msgr2=0x7fedc80fbfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:55.161 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.158+0000 7fedce30b700 1 --2- 172.21.15.39:0/2594936778 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedc80fbb30 0x7fedc80fbfa0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fedbc0049c0 tx=0x7fedbc009e30 comp rx=0 tx=0).stop 2023-06-22T17:25:55.161 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.158+0000 7fedce30b700 1 -- 172.21.15.39:0/2594936778 shutdown_connections 2023-06-22T17:25:55.161 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.158+0000 7fedce30b700 1 --2- 172.21.15.39:0/2594936778 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fedc80fc4e0 0x7fedc81088b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:55.161 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.158+0000 7fedce30b700 1 --2- 172.21.15.39:0/2594936778 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fedc8101af0 0x7fedc8101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:55.162 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.158+0000 7fedce30b700 1 --2- 172.21.15.39:0/2594936778 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedc80fbb30 0x7fedc80fbfa0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:55.162 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.158+0000 7fedce30b700 1 -- 172.21.15.39:0/2594936778 >> 172.21.15.39:0/2594936778 conn(0x7fedc8072880 msgr2=0x7fedc80f9c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:25:55.162 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.158+0000 7fedce30b700 1 -- 172.21.15.39:0/2594936778 shutdown_connections 2023-06-22T17:25:55.162 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.158+0000 7fedce30b700 1 -- 172.21.15.39:0/2594936778 wait complete. 2023-06-22T17:25:55.162 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.159+0000 7fedce30b700 1 Processor -- start 2023-06-22T17:25:55.162 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.159+0000 7fedce30b700 1 -- start start 2023-06-22T17:25:55.164 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.159+0000 7fedce30b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fedc80fbb30 0x7fedc80fd710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:25:55.164 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.159+0000 7fedce30b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fedc80fc4e0 0x7fedc80fdc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:25:55.164 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.159+0000 7fedce30b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedc8101af0 0x7fedc80fe190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:25:55.164 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.159+0000 7fedce30b700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc811c030 con 0x7fedc8101af0 2023-06-22T17:25:55.164 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.159+0000 7fedce30b700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc811c1a0 con 0x7fedc80fc4e0 2023-06-22T17:25:55.165 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.159+0000 7fedce30b700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc811c360 con 0x7fedc80fbb30 2023-06-22T17:25:55.165 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.159+0000 7fedcc8a8700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedc8101af0 0x7fedc80fe190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:25:55.165 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.159+0000 7fedc7fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fedc80fbb30 0x7fedc80fd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:25:55.165 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.159+0000 7fedcc8a8700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedc8101af0 0x7fedc80fe190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34470/0 (socket says 172.21.15.39:34470) 2023-06-22T17:25:55.165 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.159+0000 7fedcc8a8700 1 -- 172.21.15.39:0/1418151371 learned_addr learned my addr 172.21.15.39:0/1418151371 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:25:55.165 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.159+0000 7fedc77fe700 1 --2- 172.21.15.39:0/1418151371 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fedc80fc4e0 0x7fedc80fdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:25:55.165 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.160+0000 7fedcc8a8700 1 -- 172.21.15.39:0/1418151371 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fedc80fbb30 msgr2=0x7fedc80fd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:55.166 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.160+0000 7fedcc8a8700 1 --2- 172.21.15.39:0/1418151371 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fedc80fbb30 0x7fedc80fd710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:55.166 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.160+0000 7fedcc8a8700 1 -- 172.21.15.39:0/1418151371 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fedc80fc4e0 msgr2=0x7fedc80fdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:55.166 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.160+0000 7fedcc8a8700 1 --2- 172.21.15.39:0/1418151371 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fedc80fc4e0 0x7fedc80fdc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:55.166 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.160+0000 7fedcc8a8700 1 -- 172.21.15.39:0/1418151371 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedbc005040 con 0x7fedc8101af0 2023-06-22T17:25:55.166 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.160+0000 7fedcc8a8700 1 --2- 172.21.15.39:0/1418151371 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedc8101af0 0x7fedc80fe190 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fedc000cb90 tx=0x7fedc0007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:25:55.166 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.160+0000 7fedc4ff9700 1 -- 172.21.15.39:0/1418151371 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc0004e50 con 0x7fedc8101af0 2023-06-22T17:25:55.166 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.160+0000 7fedce30b700 1 -- 172.21.15.39:0/1418151371 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedc811c640 con 0x7fedc8101af0 2023-06-22T17:25:55.167 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.160+0000 7fedc4ff9700 1 -- 172.21.15.39:0/1418151371 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7fedc000ac80 con 0x7fedc8101af0 2023-06-22T17:25:55.167 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.160+0000 7fedce30b700 1 -- 172.21.15.39:0/1418151371 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedc811cab0 con 0x7fedc8101af0 2023-06-22T17:25:55.167 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.160+0000 7fedc4ff9700 1 -- 172.21.15.39:0/1418151371 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc0003970 con 0x7fedc8101af0 2023-06-22T17:25:55.167 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.161+0000 7fedce30b700 1 -- 172.21.15.39:0/1418151371 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedc804c1b0 con 0x7fedc8101af0 2023-06-22T17:25:55.167 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.162+0000 7fedc4ff9700 1 -- 172.21.15.39:0/1418151371 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7fedc0003da0 con 0x7fedc8101af0 2023-06-22T17:25:55.168 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.163+0000 7fedc4ff9700 1 --2- 172.21.15.39:0/1418151371 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fedb009b680 0x7fedb009db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:25:55.168 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.163+0000 7fedc7fff700 1 --2- 172.21.15.39:0/1418151371 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fedb009b680 0x7fedb009db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:25:55.168 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.163+0000 7fedc4ff9700 1 -- 172.21.15.39:0/1418151371 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fedc00c4ca0 con 0x7fedc8101af0 2023-06-22T17:25:55.168 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.163+0000 7fedc7fff700 1 --2- 172.21.15.39:0/1418151371 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fedb009b680 0x7fedb009db30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fedb8004f40 tx=0x7fedb800b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:25:55.169 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.167+0000 7fedc4ff9700 1 -- 172.21.15.39:0/1418151371 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fedc0090900 con 0x7fedc8101af0 2023-06-22T17:25:55.364 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.363+0000 7fedce30b700 1 -- 172.21.15.39:0/1418151371 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fedc811cd80 con 0x7fedc8101af0 2023-06-22T17:25:55.365 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.363+0000 7fedc4ff9700 1 -- 172.21.15.39:0/1418151371 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+18158 (secure 0 0 0) 0x7fedc0093f20 con 0x7fedc8101af0 2023-06-22T17:25:55.366 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:25:55.366 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":77,"fsid":"d88002e2-1120-11ee-9b2b-001a4aab830c","created":"2023-06-22T17:19:36.313754+0000","modified":"2023-06-22T17:25:52.036913+0000","last_up_change":"2023-06-22T17:25:51.023671+0000","last_in_change":"2023-06-22T17:25:33.400238+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-22T17:22:35.797853+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"eb8f5cba-9266-47ec-8af1-8e9edd6ec96a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":453150307},{"type":"v1","addr":"172.21.15.39:6803","nonce":453150307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":453150307},{"type":"v1","addr":"172.21.15.39:6805","nonce":453150307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":453150307},{"type":"v1","addr":"172.21.15.39:6809","nonce":453150307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":453150307},{"type":"v1","addr":"172.21.15.39:6807","nonce":453150307}]},"public_addr":"172.21.15.39:6803/453150307","cluster_addr":"172.21.15.39:6805/453150307","heartbeat_back_addr":"172.21.15.39:6809/453150307","heartbeat_front_addr":"172.21.15.39:6807/453150307","state":["exists","up"]},{"osd":1,"uuid":"8b8dbe3e-fd72-4f92-98a2-e212df5366c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6800","nonce":2708845767},{"type":"v1","addr":"172.21.15.101:6801","nonce":2708845767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2708845767},{"type":"v1","addr":"172.21.15.101:6803","nonce":2708845767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2708845767},{"type":"v1","addr":"172.21.15.101:6807","nonce":2708845767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2708845767},{"type":"v1","addr":"172.21.15.101:6805","nonce":2708845767}]},"public_addr":"172.21.15.101:6801/2708845767","cluster_addr":"172.21.15.101:6803/2708845767","heartbeat_back_addr":"172.21.15.101:6807/2708845767","heartbeat_front_addr":"172.21.15.101:6805/2708845767","state":["exists","up"]},{"osd":2,"uuid":"7b25ade6-b5e2-4194-92c7-3baf8c4e9560","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":3632902853},{"type":"v1","addr":"172.21.15.133:6801","nonce":3632902853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":3632902853},{"type":"v1","addr":"172.21.15.133:6803","nonce":3632902853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":3632902853},{"type":"v1","addr":"172.21.15.133:6807","nonce":3632902853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":3632902853},{"type":"v1","addr":"172.21.15.133:6805","nonce":3632902853}]},"public_addr":"172.21.15.133:6801/3632902853","cluster_addr":"172.21.15.133:6803/3632902853","heartbeat_back_addr":"172.21.15.133:6807/3632902853","heartbeat_front_addr":"172.21.15.133:6805/3632902853","state":["exists","up"]},{"osd":3,"uuid":"b91b9111-cb8f-4c51-85c6-96c0851d57d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1215819990},{"type":"v1","addr":"172.21.15.39:6811","nonce":1215819990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1215819990},{"type":"v1","addr":"172.21.15.39:6813","nonce":1215819990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":1215819990},{"type":"v1","addr":"172.21.15.39:6817","nonce":1215819990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1215819990},{"type":"v1","addr":"172.21.15.39:6815","nonce":1215819990}]},"public_addr":"172.21.15.39:6811/1215819990","cluster_addr":"172.21.15.39:6813/1215819990","heartbeat_back_addr":"172.21.15.39:6817/1215819990","heartbeat_front_addr":"172.21.15.39:6815/1215819990","state":["exists","up"]},{"osd":4,"uuid":"14f6d038-1e3c-41e0-9be6-3615099ac5fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":1371317064},{"type":"v1","addr":"172.21.15.101:6809","nonce":1371317064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":1371317064},{"type":"v1","addr":"172.21.15.101:6811","nonce":1371317064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":1371317064},{"type":"v1","addr":"172.21.15.101:6815","nonce":1371317064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":1371317064},{"type":"v1","addr":"172.21.15.101:6813","nonce":1371317064}]},"public_addr":"172.21.15.101:6809/1371317064","cluster_addr":"172.21.15.101:6811/1371317064","heartbeat_back_addr":"172.21.15.101:6815/1371317064","heartbeat_front_addr":"172.21.15.101:6813/1371317064","state":["exists","up"]},{"osd":5,"uuid":"6daaeb92-c73d-4c30-bb1d-18d5305a63a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":747181981},{"type":"v1","addr":"172.21.15.133:6809","nonce":747181981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":747181981},{"type":"v1","addr":"172.21.15.133:6811","nonce":747181981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":747181981},{"type":"v1","addr":"172.21.15.133:6815","nonce":747181981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":747181981},{"type":"v1","addr":"172.21.15.133:6813","nonce":747181981}]},"public_addr":"172.21.15.133:6809/747181981","cluster_addr":"172.21.15.133:6811/747181981","heartbeat_back_addr":"172.21.15.133:6815/747181981","heartbeat_front_addr":"172.21.15.133:6813/747181981","state":["exists","up"]},{"osd":6,"uuid":"21f46569-8123-4bdd-b3c5-4cc8bc271899","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1221216882},{"type":"v1","addr":"172.21.15.39:6819","nonce":1221216882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1221216882},{"type":"v1","addr":"172.21.15.39:6821","nonce":1221216882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1221216882},{"type":"v1","addr":"172.21.15.39:6825","nonce":1221216882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1221216882},{"type":"v1","addr":"172.21.15.39:6823","nonce":1221216882}]},"public_addr":"172.21.15.39:6819/1221216882","cluster_addr":"172.21.15.39:6821/1221216882","heartbeat_back_addr":"172.21.15.39:6825/1221216882","heartbeat_front_addr":"172.21.15.39:6823/1221216882","state":["exists","up"]},{"osd":7,"uuid":"7a1cbc1b-8cf0-4153-b8e2-3d3313bed2ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":1606613660},{"type":"v1","addr":"172.21.15.101:6817","nonce":1606613660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":1606613660},{"type":"v1","addr":"172.21.15.101:6819","nonce":1606613660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":1606613660},{"type":"v1","addr":"172.21.15.101:6823","nonce":1606613660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":1606613660},{"type":"v1","addr":"172.21.15.101:6821","nonce":1606613660}]},"public_addr":"172.21.15.101:6817/1606613660","cluster_addr":"172.21.15.101:6819/1606613660","heartbeat_back_addr":"172.21.15.101:6823/1606613660","heartbeat_front_addr":"172.21.15.101:6821/1606613660","state":["exists","up"]},{"osd":8,"uuid":"ab09136e-4326-4c02-ac59-3d2264e85b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":1363381164},{"type":"v1","addr":"172.21.15.133:6817","nonce":1363381164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":1363381164},{"type":"v1","addr":"172.21.15.133:6819","nonce":1363381164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":1363381164},{"type":"v1","addr":"172.21.15.133:6823","nonce":1363381164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":1363381164},{"type":"v1","addr":"172.21.15.133:6821","nonce":1363381164}]},"public_addr":"172.21.15.133:6817/1363381164","cluster_addr":"172.21.15.133:6819/1363381164","heartbeat_back_addr":"172.21.15.133:6823/1363381164","heartbeat_front_addr":"172.21.15.133:6821/1363381164","state":["exists","up"]},{"osd":9,"uuid":"ec8c1344-0478-4094-b544-3360054fd92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3010986736},{"type":"v1","addr":"172.21.15.39:6827","nonce":3010986736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3010986736},{"type":"v1","addr":"172.21.15.39:6829","nonce":3010986736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3010986736},{"type":"v1","addr":"172.21.15.39:6833","nonce":3010986736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3010986736},{"type":"v1","addr":"172.21.15.39:6831","nonce":3010986736}]},"public_addr":"172.21.15.39:6827/3010986736","cluster_addr":"172.21.15.39:6829/3010986736","heartbeat_back_addr":"172.21.15.39:6833/3010986736","heartbeat_front_addr":"172.21.15.39:6831/3010986736","state":["exists","up"]},{"osd":10,"uuid":"9da2943f-b947-49d6-bab1-bc3a95299996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3982067769},{"type":"v1","addr":"172.21.15.101:6825","nonce":3982067769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":3982067769},{"type":"v1","addr":"172.21.15.101:6827","nonce":3982067769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":3982067769},{"type":"v1","addr":"172.21.15.101:6831","nonce":3982067769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":3982067769},{"type":"v1","addr":"172.21.15.101:6829","nonce":3982067769}]},"public_addr":"172.21.15.101:6825/3982067769","cluster_addr":"172.21.15.101:6827/3982067769","heartbeat_back_addr":"172.21.15.101:6831/3982067769","heartbeat_front_addr":"172.21.15.101:6829/3982067769","state":["exists","up"]},{"osd":11,"uuid":"ee5d51b4-c9ad-4603-a967-c60f3c1f2cdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":1265049047},{"type":"v1","addr":"172.21.15.133:6825","nonce":1265049047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6826","nonce":1265049047},{"type":"v1","addr":"172.21.15.133:6827","nonce":1265049047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6830","nonce":1265049047},{"type":"v1","addr":"172.21.15.133:6831","nonce":1265049047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6828","nonce":1265049047},{"type":"v1","addr":"172.21.15.133:6829","nonce":1265049047}]},"public_addr":"172.21.15.133:6825/1265049047","cluster_addr":"172.21.15.133:6827/1265049047","heartbeat_back_addr":"172.21.15.133:6831/1265049047","heartbeat_front_addr":"172.21.15.133:6829/1265049047","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:21:49.604087+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:22:11.298409+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:22:32.913915+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:22:56.095316+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:23:17.505593+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:23:38.490450+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:24:01.498630+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:24:22.952502+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:24:44.480122+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:25:07.141899+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:25:28.109970+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:25:49.227776+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:6801/1350590065":"2023-06-23T17:20:27.281966+0000","172.21.15.39:6800/1350590065":"2023-06-23T17:20:27.281966+0000","172.21.15.39:0/82302526":"2023-06-23T17:20:27.281966+0000","172.21.15.39:0/260892351":"2023-06-23T17:20:05.206927+0000","172.21.15.39:0/1513096987":"2023-06-23T17:20:05.206927+0000","172.21.15.39:6800/3960151496":"2023-06-23T17:20:05.206927+0000","172.21.15.39:6801/3960151496":"2023-06-23T17:20:05.206927+0000","172.21.15.39:0/2480502158":"2023-06-23T17:20:27.281966+0000","172.21.15.39:0/3848722689":"2023-06-23T17:20:27.281966+0000","172.21.15.39:0/2257210574":"2023-06-23T17:20:05.206927+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T17:25:55.367 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.366+0000 7fedce30b700 1 -- 172.21.15.39:0/1418151371 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fedb009b680 msgr2=0x7fedb009db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:55.368 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.366+0000 7fedce30b700 1 --2- 172.21.15.39:0/1418151371 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fedb009b680 0x7fedb009db30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fedb8004f40 tx=0x7fedb800b040 comp rx=0 tx=0).stop 2023-06-22T17:25:55.368 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.366+0000 7fedce30b700 1 -- 172.21.15.39:0/1418151371 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedc8101af0 msgr2=0x7fedc80fe190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:55.368 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.366+0000 7fedce30b700 1 --2- 172.21.15.39:0/1418151371 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedc8101af0 0x7fedc80fe190 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fedc000cb90 tx=0x7fedc0007f50 comp rx=0 tx=0).stop 2023-06-22T17:25:55.368 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.366+0000 7fedce30b700 1 -- 172.21.15.39:0/1418151371 shutdown_connections 2023-06-22T17:25:55.368 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.366+0000 7fedce30b700 1 --2- 172.21.15.39:0/1418151371 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fedb009b680 0x7fedb009db30 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:55.369 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.366+0000 7fedce30b700 1 --2- 172.21.15.39:0/1418151371 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedc8101af0 0x7fedc80fe190 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:55.369 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.366+0000 7fedce30b700 1 --2- 172.21.15.39:0/1418151371 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fedc80fbb30 0x7fedc80fd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:55.369 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.366+0000 7fedce30b700 1 --2- 172.21.15.39:0/1418151371 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fedc80fc4e0 0x7fedc80fdc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:55.369 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.366+0000 7fedce30b700 1 -- 172.21.15.39:0/1418151371 >> 172.21.15.39:0/1418151371 conn(0x7fedc8072880 msgr2=0x7fedc8073ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:25:55.369 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.366+0000 7fedce30b700 1 -- 172.21.15.39:0/1418151371 shutdown_connections 2023-06-22T17:25:55.369 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:55.366+0000 7fedce30b700 1 -- 172.21.15.39:0/1418151371 wait complete. 2023-06-22T17:25:55.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:55 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/1418151371' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-06-22T17:25:55.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:55 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/1418151371' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-06-22T17:25:55.771 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-22T17:22:35.797853+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-06-22T17:25:55.772 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph osd pool get .mgr pg_num 2023-06-22T17:25:55.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:55 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/1418151371' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-06-22T17:25:56.365 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-22T17:25:56.670 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:56 smithi133 ceph-mon[86268]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:25:56.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:56 smithi039 ceph-mon[84805]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:25:56.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:56 smithi101 ceph-mon[85507]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:25:57.577 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.574+0000 7f28d0fb2700 1 -- 172.21.15.39:0/865203911 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28cc102c90 msgr2=0x7f28cc103060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:57.577 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.574+0000 7f28d0fb2700 1 --2- 172.21.15.39:0/865203911 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28cc102c90 0x7f28cc103060 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f28bc0049c0 tx=0x7f28bc009e30 comp rx=0 tx=0).stop 2023-06-22T17:25:57.577 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.574+0000 7f28d0fb2700 1 -- 172.21.15.39:0/865203911 shutdown_connections 2023-06-22T17:25:57.577 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.574+0000 7f28d0fb2700 1 --2- 172.21.15.39:0/865203911 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f28cc0fd6b0 0x7f28cc1099a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:57.577 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.574+0000 7f28d0fb2700 1 --2- 172.21.15.39:0/865203911 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28cc102c90 0x7f28cc103060 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:57.578 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.574+0000 7f28d0fb2700 1 --2- 172.21.15.39:0/865203911 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f28cc0fcd00 0x7f28cc0fd170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:57.578 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.574+0000 7f28d0fb2700 1 -- 172.21.15.39:0/865203911 >> 172.21.15.39:0/865203911 conn(0x7f28cc074af0 msgr2=0x7f28cc074ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:25:57.578 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.574+0000 7f28d0fb2700 1 -- 172.21.15.39:0/865203911 shutdown_connections 2023-06-22T17:25:57.578 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.574+0000 7f28d0fb2700 1 -- 172.21.15.39:0/865203911 wait complete. 2023-06-22T17:25:57.578 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.574+0000 7f28d0fb2700 1 Processor -- start 2023-06-22T17:25:57.578 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.574+0000 7f28d0fb2700 1 -- start start 2023-06-22T17:25:57.578 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.575+0000 7f28d0fb2700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f28cc0fcd00 0x7f28cc113900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:25:57.578 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.575+0000 7f28d0fb2700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28cc0fd6b0 0x7f28cc113e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:25:57.579 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.575+0000 7f28d0fb2700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f28cc102c90 0x7f28cc10d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:25:57.579 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.575+0000 7f28d0fb2700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28cc10e040 con 0x7f28cc0fd6b0 2023-06-22T17:25:57.579 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.575+0000 7f28d0fb2700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28cc10e1b0 con 0x7f28cc0fcd00 2023-06-22T17:25:57.579 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.575+0000 7f28d0fb2700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28cc10e370 con 0x7f28cc102c90 2023-06-22T17:25:57.579 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.575+0000 7f28c9d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28cc0fd6b0 0x7f28cc113e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:25:57.579 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.575+0000 7f28c9d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28cc0fd6b0 0x7f28cc113e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48880/0 (socket says 172.21.15.39:48880) 2023-06-22T17:25:57.579 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.575+0000 7f28c9d9b700 1 -- 172.21.15.39:0/3010612997 learned_addr learned my addr 172.21.15.39:0/3010612997 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:25:57.580 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.575+0000 7f28ca59c700 1 --2- 172.21.15.39:0/3010612997 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f28cc0fcd00 0x7f28cc113900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:25:57.580 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.575+0000 7f28cad9d700 1 --2- 172.21.15.39:0/3010612997 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f28cc102c90 0x7f28cc10d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:25:57.580 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.575+0000 7f28c9d9b700 1 -- 172.21.15.39:0/3010612997 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f28cc102c90 msgr2=0x7f28cc10d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:57.580 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.575+0000 7f28c9d9b700 1 --2- 172.21.15.39:0/3010612997 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f28cc102c90 0x7f28cc10d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:57.580 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.575+0000 7f28c9d9b700 1 -- 172.21.15.39:0/3010612997 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f28cc0fcd00 msgr2=0x7f28cc113900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:57.580 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.575+0000 7f28c9d9b700 1 --2- 172.21.15.39:0/3010612997 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f28cc0fcd00 0x7f28cc113900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:57.580 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.575+0000 7f28c9d9b700 1 -- 172.21.15.39:0/3010612997 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28bc005040 con 0x7f28cc0fd6b0 2023-06-22T17:25:57.581 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.575+0000 7f28c9d9b700 1 --2- 172.21.15.39:0/3010612997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28cc0fd6b0 0x7f28cc113e40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f28c0002b00 tx=0x7f28c00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:25:57.582 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.575+0000 7f28baffd700 1 -- 172.21.15.39:0/3010612997 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c0007cc0 con 0x7f28cc0fd6b0 2023-06-22T17:25:57.582 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.575+0000 7f28baffd700 1 -- 172.21.15.39:0/3010612997 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f28c000f040 con 0x7f28cc0fd6b0 2023-06-22T17:25:57.582 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.575+0000 7f28d0fb2700 1 -- 172.21.15.39:0/3010612997 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28cc10e650 con 0x7f28cc0fd6b0 2023-06-22T17:25:57.582 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.575+0000 7f28baffd700 1 -- 172.21.15.39:0/3010612997 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c000da30 con 0x7f28cc0fd6b0 2023-06-22T17:25:57.582 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.576+0000 7f28d0fb2700 1 -- 172.21.15.39:0/3010612997 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28cc11a8c0 con 0x7f28cc0fd6b0 2023-06-22T17:25:57.582 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.577+0000 7f28d0fb2700 1 -- 172.21.15.39:0/3010612997 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28cc04c1b0 con 0x7f28cc0fd6b0 2023-06-22T17:25:57.583 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.578+0000 7f28baffd700 1 -- 172.21.15.39:0/3010612997 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f28c0009090 con 0x7f28cc0fd6b0 2023-06-22T17:25:57.583 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.578+0000 7f28baffd700 1 --2- 172.21.15.39:0/3010612997 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f28b409b700 0x7f28b409dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:25:57.583 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.579+0000 7f28ca59c700 1 --2- 172.21.15.39:0/3010612997 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f28b409b700 0x7f28b409dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:25:57.583 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.579+0000 7f28baffd700 1 -- 172.21.15.39:0/3010612997 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f28c00c4a80 con 0x7f28cc0fd6b0 2023-06-22T17:25:57.583 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.579+0000 7f28ca59c700 1 --2- 172.21.15.39:0/3010612997 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f28b409b700 0x7f28b409dbb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f28bc005010 tx=0x7f28bc009e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:25:57.584 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.582+0000 7f28baffd700 1 -- 172.21.15.39:0/3010612997 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f28c0090760 con 0x7f28cc0fd6b0 2023-06-22T17:25:57.775 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.773+0000 7f28d0fb2700 1 -- 172.21.15.39:0/3010612997 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f28cc11aae0 con 0x7f28cc0fd6b0 2023-06-22T17:25:57.775 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.773+0000 7f28baffd700 1 -- 172.21.15.39:0/3010612997 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7f28c000de60 con 0x7f28cc0fd6b0 2023-06-22T17:25:57.776 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2023-06-22T17:25:57.777 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.775+0000 7f28d0fb2700 1 -- 172.21.15.39:0/3010612997 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f28b409b700 msgr2=0x7f28b409dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:57.777 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.775+0000 7f28d0fb2700 1 --2- 172.21.15.39:0/3010612997 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f28b409b700 0x7f28b409dbb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f28bc005010 tx=0x7f28bc009e30 comp rx=0 tx=0).stop 2023-06-22T17:25:57.777 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.775+0000 7f28d0fb2700 1 -- 172.21.15.39:0/3010612997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28cc0fd6b0 msgr2=0x7f28cc113e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:57.777 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.775+0000 7f28d0fb2700 1 --2- 172.21.15.39:0/3010612997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28cc0fd6b0 0x7f28cc113e40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f28c0002b00 tx=0x7f28c00046b0 comp rx=0 tx=0).stop 2023-06-22T17:25:57.778 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.775+0000 7f28d0fb2700 1 -- 172.21.15.39:0/3010612997 shutdown_connections 2023-06-22T17:25:57.778 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.775+0000 7f28d0fb2700 1 --2- 172.21.15.39:0/3010612997 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f28b409b700 0x7f28b409dbb0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:57.778 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.775+0000 7f28d0fb2700 1 --2- 172.21.15.39:0/3010612997 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f28cc102c90 0x7f28cc10d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:57.778 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.775+0000 7f28d0fb2700 1 --2- 172.21.15.39:0/3010612997 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f28cc0fcd00 0x7f28cc113900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:57.778 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.775+0000 7f28d0fb2700 1 --2- 172.21.15.39:0/3010612997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28cc0fd6b0 0x7f28cc113e40 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:57.778 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.775+0000 7f28d0fb2700 1 -- 172.21.15.39:0/3010612997 >> 172.21.15.39:0/3010612997 conn(0x7f28cc074af0 msgr2=0x7f28cc1039c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:25:57.778 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.775+0000 7f28d0fb2700 1 -- 172.21.15.39:0/3010612997 shutdown_connections 2023-06-22T17:25:57.779 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:25:57.775+0000 7f28d0fb2700 1 -- 172.21.15.39:0/3010612997 wait complete. 2023-06-22T17:25:58.204 INFO:tasks.cephadm:Adding mds.a on smithi039 2023-06-22T17:25:58.204 INFO:tasks.cephadm:Adding mds.d on smithi039 2023-06-22T17:25:58.204 INFO:tasks.cephadm:Adding mds.g on smithi039 2023-06-22T17:25:58.205 INFO:tasks.cephadm:Adding mds.j on smithi039 2023-06-22T17:25:58.205 INFO:tasks.cephadm:Adding mds.b on smithi101 2023-06-22T17:25:58.205 INFO:tasks.cephadm:Adding mds.e on smithi101 2023-06-22T17:25:58.205 INFO:tasks.cephadm:Adding mds.h on smithi101 2023-06-22T17:25:58.205 INFO:tasks.cephadm:Adding mds.k on smithi101 2023-06-22T17:25:58.206 INFO:tasks.cephadm:Adding mds.c on smithi133 2023-06-22T17:25:58.206 INFO:tasks.cephadm:Adding mds.f on smithi133 2023-06-22T17:25:58.206 INFO:tasks.cephadm:Adding mds.i on smithi133 2023-06-22T17:25:58.206 INFO:tasks.cephadm:Adding mds.l on smithi133 2023-06-22T17:25:58.206 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d88002e2-1120-11ee-9b2b-001a4aab830c -- ceph orch apply mds all '12;smithi039=a;smithi039=d;smithi039=g;smithi039=j;smithi101=b;smithi101=e;smithi101=h;smithi101=k;smithi133=c;smithi133=f;smithi133=i;smithi133=l' 2023-06-22T17:25:58.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:58 smithi133 ceph-mon[86268]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:25:58.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:25:58 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/3010612997' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-06-22T17:25:58.802 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/mon.c/config 2023-06-22T17:25:58.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:58 smithi101 ceph-mon[85507]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:25:58.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:25:58 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/3010612997' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-06-22T17:25:58.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:58 smithi039 ceph-mon[84805]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:25:58.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:25:58 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/3010612997' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-06-22T17:25:59.944 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.941+0000 7fb162cfe700 1 -- 172.21.15.133:0/2932153 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb15c0fec90 msgr2=0x7fb15c0fad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:59.944 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.941+0000 7fb162cfe700 1 --2- 172.21.15.133:0/2932153 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb15c0fec90 0x7fb15c0fad40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb1480049c0 tx=0x7fb148009e30 comp rx=0 tx=0).stop 2023-06-22T17:25:59.945 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.942+0000 7fb162cfe700 1 -- 172.21.15.133:0/2932153 shutdown_connections 2023-06-22T17:25:59.945 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.942+0000 7fb162cfe700 1 --2- 172.21.15.133:0/2932153 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb15c0fb3b0 0x7fb15c1052b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:59.945 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.942+0000 7fb162cfe700 1 --2- 172.21.15.133:0/2932153 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb15c0fe2b0 0x7fb15c0fe6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:59.945 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.942+0000 7fb162cfe700 1 --2- 172.21.15.133:0/2932153 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb15c0fec90 0x7fb15c0fad40 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:59.945 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.942+0000 7fb162cfe700 1 -- 172.21.15.133:0/2932153 >> 172.21.15.133:0/2932153 conn(0x7fb15c0f6a10 msgr2=0x7fb15c0f8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:25:59.945 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.942+0000 7fb162cfe700 1 -- 172.21.15.133:0/2932153 shutdown_connections 2023-06-22T17:25:59.945 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.942+0000 7fb162cfe700 1 -- 172.21.15.133:0/2932153 wait complete. 2023-06-22T17:25:59.946 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.942+0000 7fb162cfe700 1 Processor -- start 2023-06-22T17:25:59.946 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.942+0000 7fb162cfe700 1 -- start start 2023-06-22T17:25:59.947 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.942+0000 7fb162cfe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb15c0fb3b0 0x7fb15c0fd910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:25:59.947 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.942+0000 7fb162cfe700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb15c0fe2b0 0x7fb15c0fbf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:25:59.948 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.942+0000 7fb162cfe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb15c0fc4b0 0x7fb15c0fc950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:25:59.948 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.942+0000 7fb162cfe700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb15c117be0 con 0x7fb15c0fb3b0 2023-06-22T17:25:59.948 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.942+0000 7fb162cfe700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb15c117d50 con 0x7fb15c0fe2b0 2023-06-22T17:25:59.948 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.942+0000 7fb162cfe700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb15c117f10 con 0x7fb15c0fc4b0 2023-06-22T17:25:59.949 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.943+0000 7fb16129b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb15c0fc4b0 0x7fb15c0fc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:25:59.949 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.943+0000 7fb16129b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb15c0fc4b0 0x7fb15c0fc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:54330/0 (socket says 172.21.15.133:54330) 2023-06-22T17:25:59.949 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.943+0000 7fb16129b700 1 -- 172.21.15.133:0/555284879 learned_addr learned my addr 172.21.15.133:0/555284879 (peer_addr_for_me v2:172.21.15.133:0/0) 2023-06-22T17:25:59.950 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.943+0000 7fb160a9a700 1 --2- 172.21.15.133:0/555284879 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb15c0fb3b0 0x7fb15c0fd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:25:59.950 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.943+0000 7fb15bfff700 1 --2- 172.21.15.133:0/555284879 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb15c0fe2b0 0x7fb15c0fbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:25:59.950 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.943+0000 7fb16129b700 1 -- 172.21.15.133:0/555284879 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb15c0fe2b0 msgr2=0x7fb15c0fbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:59.950 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.943+0000 7fb16129b700 1 --2- 172.21.15.133:0/555284879 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb15c0fe2b0 0x7fb15c0fbf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:59.951 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.943+0000 7fb16129b700 1 -- 172.21.15.133:0/555284879 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb15c0fb3b0 msgr2=0x7fb15c0fd910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:25:59.951 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.943+0000 7fb16129b700 1 --2- 172.21.15.133:0/555284879 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb15c0fb3b0 0x7fb15c0fd910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:25:59.951 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.943+0000 7fb16129b700 1 -- 172.21.15.133:0/555284879 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb148005040 con 0x7fb15c0fc4b0 2023-06-22T17:25:59.951 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.943+0000 7fb15bfff700 1 --2- 172.21.15.133:0/555284879 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb15c0fe2b0 0x7fb15c0fbf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T17:25:59.952 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.943+0000 7fb16129b700 1 --2- 172.21.15.133:0/555284879 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb15c0fc4b0 0x7fb15c0fc950 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb15400bc30 tx=0x7fb154009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:25:59.952 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.943+0000 7fb1597fa700 1 -- 172.21.15.133:0/555284879 <== mon.1 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1540092e0 con 0x7fb15c0fc4b0 2023-06-22T17:25:59.952 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.943+0000 7fb1597fa700 1 -- 172.21.15.133:0/555284879 <== mon.1 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7fb15400cbd0 con 0x7fb15c0fc4b0 2023-06-22T17:25:59.952 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.943+0000 7fb162cfe700 1 -- 172.21.15.133:0/555284879 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb15c1181f0 con 0x7fb15c0fc4b0 2023-06-22T17:25:59.953 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.943+0000 7fb1597fa700 1 -- 172.21.15.133:0/555284879 <== mon.1 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb154015d20 con 0x7fb15c0fc4b0 2023-06-22T17:25:59.953 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.943+0000 7fb162cfe700 1 -- 172.21.15.133:0/555284879 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb15c1186e0 con 0x7fb15c0fc4b0 2023-06-22T17:25:59.953 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.944+0000 7fb162cfe700 1 -- 172.21.15.133:0/555284879 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb15c103cc0 con 0x7fb15c0fc4b0 2023-06-22T17:25:59.953 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.945+0000 7fb1597fa700 1 -- 172.21.15.133:0/555284879 <== mon.1 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7fb154012070 con 0x7fb15c0fc4b0 2023-06-22T17:25:59.955 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.946+0000 7fb1597fa700 1 --2- 172.21.15.133:0/555284879 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fb14c09b680 0x7fb14c09db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:25:59.955 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.946+0000 7fb1597fa700 1 -- 172.21.15.133:0/555284879 <== mon.1 v2:172.21.15.133:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fb1540c48a0 con 0x7fb15c0fc4b0 2023-06-22T17:25:59.955 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.946+0000 7fb160a9a700 1 --2- 172.21.15.133:0/555284879 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fb14c09b680 0x7fb14c09db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:25:59.955 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.947+0000 7fb160a9a700 1 --2- 172.21.15.133:0/555284879 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fb14c09b680 0x7fb14c09db30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb15c065c60 tx=0x7fb150009040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:25:59.955 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:25:59.949+0000 7fb1597fa700 1 -- 172.21.15.133:0/555284879 <== mon.1 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fb154090500 con 0x7fb15c0fc4b0 2023-06-22T17:26:00.144 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:26:00.142+0000 7fb162cfe700 1 -- 172.21.15.133:0/555284879 --> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi039=a;smithi039=d;smithi039=g;smithi039=j;smithi101=b;smithi101=e;smithi101=h;smithi101=k;smithi133=c;smithi133=f;smithi133=i;smithi133=l", "target": ["mon-mgr", ""]}) v1 -- 0x7fb15c05de70 con 0x7fb14c09b680 2023-06-22T17:26:00.153 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:26:00.151+0000 7fb1597fa700 1 -- 172.21.15.133:0/555284879 <== mgr.14150 v2:172.21.15.39:6800/886078282 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fb15c05de70 con 0x7fb14c09b680 2023-06-22T17:26:00.154 INFO:teuthology.orchestra.run.smithi133.stdout:Scheduled mds.all update... 2023-06-22T17:26:00.156 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:26:00.153+0000 7fb162cfe700 1 -- 172.21.15.133:0/555284879 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fb14c09b680 msgr2=0x7fb14c09db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:00.156 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:26:00.153+0000 7fb162cfe700 1 --2- 172.21.15.133:0/555284879 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fb14c09b680 0x7fb14c09db30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb15c065c60 tx=0x7fb150009040 comp rx=0 tx=0).stop 2023-06-22T17:26:00.156 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:26:00.153+0000 7fb162cfe700 1 -- 172.21.15.133:0/555284879 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb15c0fc4b0 msgr2=0x7fb15c0fc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:00.156 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:26:00.153+0000 7fb162cfe700 1 --2- 172.21.15.133:0/555284879 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb15c0fc4b0 0x7fb15c0fc950 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb15400bc30 tx=0x7fb154009840 comp rx=0 tx=0).stop 2023-06-22T17:26:00.156 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:26:00.154+0000 7fb162cfe700 1 -- 172.21.15.133:0/555284879 shutdown_connections 2023-06-22T17:26:00.156 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:26:00.154+0000 7fb162cfe700 1 --2- 172.21.15.133:0/555284879 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fb14c09b680 0x7fb14c09db30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:00.157 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:26:00.154+0000 7fb162cfe700 1 --2- 172.21.15.133:0/555284879 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb15c0fc4b0 0x7fb15c0fc950 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:00.157 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:26:00.154+0000 7fb162cfe700 1 --2- 172.21.15.133:0/555284879 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb15c0fb3b0 0x7fb15c0fd910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:00.157 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:26:00.154+0000 7fb162cfe700 1 --2- 172.21.15.133:0/555284879 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb15c0fe2b0 0x7fb15c0fbf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:00.157 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:26:00.154+0000 7fb162cfe700 1 -- 172.21.15.133:0/555284879 >> 172.21.15.133:0/555284879 conn(0x7fb15c0f6a10 msgr2=0x7fb15c0f8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:00.157 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:26:00.154+0000 7fb162cfe700 1 -- 172.21.15.133:0/555284879 shutdown_connections 2023-06-22T17:26:00.157 INFO:teuthology.orchestra.run.smithi133.stderr:2023-06-22T17:26:00.154+0000 7fb162cfe700 1 -- 172.21.15.133:0/555284879 wait complete. 2023-06-22T17:26:00.534 DEBUG:teuthology.orchestra.run.smithi039:mds.a> sudo journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.a.service 2023-06-22T17:26:00.536 DEBUG:teuthology.orchestra.run.smithi039:mds.d> sudo journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.d.service 2023-06-22T17:26:00.572 INFO:journalctl@ceph.mds.a.smithi039.stdout:-- Logs begin at Thu 2023-06-22 17:07:05 UTC. -- 2023-06-22T17:26:00.573 DEBUG:teuthology.orchestra.run.smithi039:mds.g> sudo journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.g.service 2023-06-22T17:26:00.601 INFO:journalctl@ceph.mds.d.smithi039.stdout:-- Logs begin at Thu 2023-06-22 17:07:05 UTC. -- 2023-06-22T17:26:00.602 DEBUG:teuthology.orchestra.run.smithi039:mds.j> sudo journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.j.service 2023-06-22T17:26:00.628 INFO:journalctl@ceph.mds.g.smithi039.stdout:-- Logs begin at Thu 2023-06-22 17:07:05 UTC. -- 2023-06-22T17:26:00.629 DEBUG:teuthology.orchestra.run.smithi101:mds.b> sudo journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.b.service 2023-06-22T17:26:00.631 DEBUG:teuthology.orchestra.run.smithi101:mds.e> sudo journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.e.service 2023-06-22T17:26:00.655 INFO:journalctl@ceph.mds.j.smithi039.stdout:-- Logs begin at Thu 2023-06-22 17:07:05 UTC. -- 2023-06-22T17:26:00.667 INFO:journalctl@ceph.mds.b.smithi101.stdout:-- Logs begin at Thu 2023-06-22 17:06:34 UTC. -- 2023-06-22T17:26:00.668 DEBUG:teuthology.orchestra.run.smithi101:mds.h> sudo journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.h.service 2023-06-22T17:26:00.695 INFO:journalctl@ceph.mds.e.smithi101.stdout:-- Logs begin at Thu 2023-06-22 17:06:34 UTC. -- 2023-06-22T17:26:00.696 DEBUG:teuthology.orchestra.run.smithi101:mds.k> sudo journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.k.service 2023-06-22T17:26:00.716 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:00 smithi101 ceph-mon[85507]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:00.716 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:00 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:00.716 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:00 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:26:00.717 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:00 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:00.717 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:00 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:26:00.717 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:00 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:00.717 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:00 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:00.717 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:00 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:00.717 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:00 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:00.718 DEBUG:teuthology.orchestra.run.smithi133:mds.c> sudo journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.c.service 2023-06-22T17:26:00.720 DEBUG:teuthology.orchestra.run.smithi133:mds.f> sudo journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.f.service 2023-06-22T17:26:00.721 INFO:journalctl@ceph.mds.h.smithi101.stdout:-- Logs begin at Thu 2023-06-22 17:06:34 UTC. -- 2023-06-22T17:26:00.741 INFO:journalctl@ceph.mds.k.smithi101.stdout:-- Logs begin at Thu 2023-06-22 17:06:34 UTC. -- 2023-06-22T17:26:00.743 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:00 smithi133 ceph-mon[86268]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:00.743 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:00 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:00.743 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:00 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:26:00.743 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:00 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:00.743 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:00 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:26:00.743 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:00 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:00.744 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:00 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:00.744 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:00 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:00.744 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:00 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:00.745 DEBUG:teuthology.orchestra.run.smithi133:mds.i> sudo journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.i.service 2023-06-22T17:26:00.750 INFO:journalctl@ceph.mds.c.smithi133.stdout:-- Logs begin at Thu 2023-06-22 17:06:34 UTC. -- 2023-06-22T17:26:00.751 DEBUG:teuthology.orchestra.run.smithi133:mds.l> sudo journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.l.service 2023-06-22T17:26:00.772 INFO:journalctl@ceph.mds.f.smithi133.stdout:-- Logs begin at Thu 2023-06-22 17:06:34 UTC. -- 2023-06-22T17:26:00.773 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2023-06-22T17:26:00.773 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-22T17:26:00.776 INFO:journalctl@ceph.mds.i.smithi133.stdout:-- Logs begin at Thu 2023-06-22 17:06:34 UTC. -- 2023-06-22T17:26:00.796 INFO:journalctl@ceph.mds.l.smithi133.stdout:-- Logs begin at Thu 2023-06-22 17:06:34 UTC. -- 2023-06-22T17:26:00.796 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:00 smithi039 ceph-mon[84805]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:00.797 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:00 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:00.797 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:00 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:26:00.797 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:00 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:00.797 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:00 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:26:00.797 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:00 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:00.797 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:00 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:00.797 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:00 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:00.798 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:00 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:00.907 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.905+0000 7f0932a7f700 1 -- 172.21.15.39:0/2022266310 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f092c0fce70 msgr2=0x7f092c0fd2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:00.907 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.905+0000 7f0932a7f700 1 --2- 172.21.15.39:0/2022266310 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f092c0fce70 0x7f092c0fd2e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f09180049c0 tx=0x7f0918009e30 comp rx=0 tx=0).stop 2023-06-22T17:26:00.907 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.905+0000 7f0932a7f700 1 -- 172.21.15.39:0/2022266310 shutdown_connections 2023-06-22T17:26:00.907 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.905+0000 7f0932a7f700 1 --2- 172.21.15.39:0/2022266310 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f092c0fd820 0x7f092c1040c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:00.908 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.905+0000 7f0932a7f700 1 --2- 172.21.15.39:0/2022266310 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f092c0fbbc0 0x7f092c0fbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:00.908 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.905+0000 7f0932a7f700 1 --2- 172.21.15.39:0/2022266310 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f092c0fce70 0x7f092c0fd2e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:00.908 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.905+0000 7f0932a7f700 1 -- 172.21.15.39:0/2022266310 >> 172.21.15.39:0/2022266310 conn(0x7f092c0f7300 msgr2=0x7f092c0f9750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:00.908 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.905+0000 7f0932a7f700 1 -- 172.21.15.39:0/2022266310 shutdown_connections 2023-06-22T17:26:00.908 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.905+0000 7f0932a7f700 1 -- 172.21.15.39:0/2022266310 wait complete. 2023-06-22T17:26:00.908 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.905+0000 7f0932a7f700 1 Processor -- start 2023-06-22T17:26:00.908 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.905+0000 7f0932a7f700 1 -- start start 2023-06-22T17:26:00.910 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.906+0000 7f0932a7f700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f092c0fbbc0 0x7f092c10c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:00.910 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.906+0000 7f0932a7f700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f092c0fce70 0x7f092c10cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:00.910 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.906+0000 7f0932a7f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f092c0fd820 0x7f092c113f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:00.911 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.906+0000 7f0932a7f700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092c1145a0 con 0x7f092c0fbbc0 2023-06-22T17:26:00.911 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.906+0000 7f0932a7f700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092c114710 con 0x7f092c0fce70 2023-06-22T17:26:00.911 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.906+0000 7f0932a7f700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092c1148d0 con 0x7f092c0fd820 2023-06-22T17:26:00.911 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.906+0000 7f093081b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f092c0fbbc0 0x7f092c10c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:00.911 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.906+0000 7f093081b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f092c0fbbc0 0x7f092c10c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48896/0 (socket says 172.21.15.39:48896) 2023-06-22T17:26:00.911 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.906+0000 7f093081b700 1 -- 172.21.15.39:0/232191767 learned_addr learned my addr 172.21.15.39:0/232191767 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:26:00.911 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.906+0000 7f093101c700 1 --2- 172.21.15.39:0/232191767 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f092c0fd820 0x7f092c113f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:00.912 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.906+0000 7f092bfff700 1 --2- 172.21.15.39:0/232191767 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f092c0fce70 0x7f092c10cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:00.912 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.906+0000 7f093081b700 1 -- 172.21.15.39:0/232191767 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f092c0fd820 msgr2=0x7f092c113f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:00.912 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.906+0000 7f093081b700 1 --2- 172.21.15.39:0/232191767 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f092c0fd820 0x7f092c113f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:00.912 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.906+0000 7f093081b700 1 -- 172.21.15.39:0/232191767 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f092c0fce70 msgr2=0x7f092c10cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:00.912 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.906+0000 7f093081b700 1 --2- 172.21.15.39:0/232191767 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f092c0fce70 0x7f092c10cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:00.912 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.906+0000 7f093081b700 1 -- 172.21.15.39:0/232191767 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0918005040 con 0x7f092c0fbbc0 2023-06-22T17:26:00.912 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.906+0000 7f093081b700 1 --2- 172.21.15.39:0/232191767 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f092c0fbbc0 0x7f092c10c9c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f092000b810 tx=0x7f0920009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:00.913 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.906+0000 7f09297fa700 1 -- 172.21.15.39:0/232191767 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092000f510 con 0x7f092c0fbbc0 2023-06-22T17:26:00.913 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.906+0000 7f09297fa700 1 -- 172.21.15.39:0/232191767 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f092000fb50 con 0x7f092c0fbbc0 2023-06-22T17:26:00.913 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.906+0000 7f0932a7f700 1 -- 172.21.15.39:0/232191767 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f092c114bb0 con 0x7f092c0fbbc0 2023-06-22T17:26:00.913 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.906+0000 7f09297fa700 1 -- 172.21.15.39:0/232191767 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0920011cd0 con 0x7f092c0fbbc0 2023-06-22T17:26:00.913 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.906+0000 7f0932a7f700 1 -- 172.21.15.39:0/232191767 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f092c1150a0 con 0x7f092c0fbbc0 2023-06-22T17:26:00.913 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.907+0000 7f0932a7f700 1 -- 172.21.15.39:0/232191767 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f092c102a80 con 0x7f092c0fbbc0 2023-06-22T17:26:00.913 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.908+0000 7f09297fa700 1 -- 172.21.15.39:0/232191767 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f092000fcc0 con 0x7f092c0fbbc0 2023-06-22T17:26:00.914 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.909+0000 7f09297fa700 1 --2- 172.21.15.39:0/232191767 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f091c09b680 0x7f091c09db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:00.914 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.909+0000 7f09297fa700 1 -- 172.21.15.39:0/232191767 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f09200c4610 con 0x7f092c0fbbc0 2023-06-22T17:26:00.914 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.909+0000 7f092bfff700 1 --2- 172.21.15.39:0/232191767 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f091c09b680 0x7f091c09db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:00.914 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.909+0000 7f092bfff700 1 --2- 172.21.15.39:0/232191767 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f091c09b680 0x7f091c09db30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0918004890 tx=0x7f0918009e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:00.915 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:00.913+0000 7f09297fa700 1 -- 172.21.15.39:0/232191767 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f0920090270 con 0x7f092c0fbbc0 2023-06-22T17:26:01.111 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.110+0000 7f0932a7f700 1 -- 172.21.15.39:0/232191767 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f092c1075e0 con 0x7f092c0fbbc0 2023-06-22T17:26:01.112 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.111+0000 7f09297fa700 1 -- 172.21.15.39:0/232191767 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+18158 (secure 0 0 0) 0x7f0920093890 con 0x7f092c0fbbc0 2023-06-22T17:26:01.113 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:26:01.113 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":77,"fsid":"d88002e2-1120-11ee-9b2b-001a4aab830c","created":"2023-06-22T17:19:36.313754+0000","modified":"2023-06-22T17:25:52.036913+0000","last_up_change":"2023-06-22T17:25:51.023671+0000","last_in_change":"2023-06-22T17:25:33.400238+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-22T17:22:35.797853+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"eb8f5cba-9266-47ec-8af1-8e9edd6ec96a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":453150307},{"type":"v1","addr":"172.21.15.39:6803","nonce":453150307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":453150307},{"type":"v1","addr":"172.21.15.39:6805","nonce":453150307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":453150307},{"type":"v1","addr":"172.21.15.39:6809","nonce":453150307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":453150307},{"type":"v1","addr":"172.21.15.39:6807","nonce":453150307}]},"public_addr":"172.21.15.39:6803/453150307","cluster_addr":"172.21.15.39:6805/453150307","heartbeat_back_addr":"172.21.15.39:6809/453150307","heartbeat_front_addr":"172.21.15.39:6807/453150307","state":["exists","up"]},{"osd":1,"uuid":"8b8dbe3e-fd72-4f92-98a2-e212df5366c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6800","nonce":2708845767},{"type":"v1","addr":"172.21.15.101:6801","nonce":2708845767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2708845767},{"type":"v1","addr":"172.21.15.101:6803","nonce":2708845767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2708845767},{"type":"v1","addr":"172.21.15.101:6807","nonce":2708845767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2708845767},{"type":"v1","addr":"172.21.15.101:6805","nonce":2708845767}]},"public_addr":"172.21.15.101:6801/2708845767","cluster_addr":"172.21.15.101:6803/2708845767","heartbeat_back_addr":"172.21.15.101:6807/2708845767","heartbeat_front_addr":"172.21.15.101:6805/2708845767","state":["exists","up"]},{"osd":2,"uuid":"7b25ade6-b5e2-4194-92c7-3baf8c4e9560","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":3632902853},{"type":"v1","addr":"172.21.15.133:6801","nonce":3632902853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":3632902853},{"type":"v1","addr":"172.21.15.133:6803","nonce":3632902853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":3632902853},{"type":"v1","addr":"172.21.15.133:6807","nonce":3632902853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":3632902853},{"type":"v1","addr":"172.21.15.133:6805","nonce":3632902853}]},"public_addr":"172.21.15.133:6801/3632902853","cluster_addr":"172.21.15.133:6803/3632902853","heartbeat_back_addr":"172.21.15.133:6807/3632902853","heartbeat_front_addr":"172.21.15.133:6805/3632902853","state":["exists","up"]},{"osd":3,"uuid":"b91b9111-cb8f-4c51-85c6-96c0851d57d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1215819990},{"type":"v1","addr":"172.21.15.39:6811","nonce":1215819990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1215819990},{"type":"v1","addr":"172.21.15.39:6813","nonce":1215819990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":1215819990},{"type":"v1","addr":"172.21.15.39:6817","nonce":1215819990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1215819990},{"type":"v1","addr":"172.21.15.39:6815","nonce":1215819990}]},"public_addr":"172.21.15.39:6811/1215819990","cluster_addr":"172.21.15.39:6813/1215819990","heartbeat_back_addr":"172.21.15.39:6817/1215819990","heartbeat_front_addr":"172.21.15.39:6815/1215819990","state":["exists","up"]},{"osd":4,"uuid":"14f6d038-1e3c-41e0-9be6-3615099ac5fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":1371317064},{"type":"v1","addr":"172.21.15.101:6809","nonce":1371317064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":1371317064},{"type":"v1","addr":"172.21.15.101:6811","nonce":1371317064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":1371317064},{"type":"v1","addr":"172.21.15.101:6815","nonce":1371317064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":1371317064},{"type":"v1","addr":"172.21.15.101:6813","nonce":1371317064}]},"public_addr":"172.21.15.101:6809/1371317064","cluster_addr":"172.21.15.101:6811/1371317064","heartbeat_back_addr":"172.21.15.101:6815/1371317064","heartbeat_front_addr":"172.21.15.101:6813/1371317064","state":["exists","up"]},{"osd":5,"uuid":"6daaeb92-c73d-4c30-bb1d-18d5305a63a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":747181981},{"type":"v1","addr":"172.21.15.133:6809","nonce":747181981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":747181981},{"type":"v1","addr":"172.21.15.133:6811","nonce":747181981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":747181981},{"type":"v1","addr":"172.21.15.133:6815","nonce":747181981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":747181981},{"type":"v1","addr":"172.21.15.133:6813","nonce":747181981}]},"public_addr":"172.21.15.133:6809/747181981","cluster_addr":"172.21.15.133:6811/747181981","heartbeat_back_addr":"172.21.15.133:6815/747181981","heartbeat_front_addr":"172.21.15.133:6813/747181981","state":["exists","up"]},{"osd":6,"uuid":"21f46569-8123-4bdd-b3c5-4cc8bc271899","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1221216882},{"type":"v1","addr":"172.21.15.39:6819","nonce":1221216882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1221216882},{"type":"v1","addr":"172.21.15.39:6821","nonce":1221216882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1221216882},{"type":"v1","addr":"172.21.15.39:6825","nonce":1221216882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1221216882},{"type":"v1","addr":"172.21.15.39:6823","nonce":1221216882}]},"public_addr":"172.21.15.39:6819/1221216882","cluster_addr":"172.21.15.39:6821/1221216882","heartbeat_back_addr":"172.21.15.39:6825/1221216882","heartbeat_front_addr":"172.21.15.39:6823/1221216882","state":["exists","up"]},{"osd":7,"uuid":"7a1cbc1b-8cf0-4153-b8e2-3d3313bed2ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":1606613660},{"type":"v1","addr":"172.21.15.101:6817","nonce":1606613660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":1606613660},{"type":"v1","addr":"172.21.15.101:6819","nonce":1606613660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":1606613660},{"type":"v1","addr":"172.21.15.101:6823","nonce":1606613660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":1606613660},{"type":"v1","addr":"172.21.15.101:6821","nonce":1606613660}]},"public_addr":"172.21.15.101:6817/1606613660","cluster_addr":"172.21.15.101:6819/1606613660","heartbeat_back_addr":"172.21.15.101:6823/1606613660","heartbeat_front_addr":"172.21.15.101:6821/1606613660","state":["exists","up"]},{"osd":8,"uuid":"ab09136e-4326-4c02-ac59-3d2264e85b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":1363381164},{"type":"v1","addr":"172.21.15.133:6817","nonce":1363381164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":1363381164},{"type":"v1","addr":"172.21.15.133:6819","nonce":1363381164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":1363381164},{"type":"v1","addr":"172.21.15.133:6823","nonce":1363381164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":1363381164},{"type":"v1","addr":"172.21.15.133:6821","nonce":1363381164}]},"public_addr":"172.21.15.133:6817/1363381164","cluster_addr":"172.21.15.133:6819/1363381164","heartbeat_back_addr":"172.21.15.133:6823/1363381164","heartbeat_front_addr":"172.21.15.133:6821/1363381164","state":["exists","up"]},{"osd":9,"uuid":"ec8c1344-0478-4094-b544-3360054fd92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3010986736},{"type":"v1","addr":"172.21.15.39:6827","nonce":3010986736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3010986736},{"type":"v1","addr":"172.21.15.39:6829","nonce":3010986736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3010986736},{"type":"v1","addr":"172.21.15.39:6833","nonce":3010986736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3010986736},{"type":"v1","addr":"172.21.15.39:6831","nonce":3010986736}]},"public_addr":"172.21.15.39:6827/3010986736","cluster_addr":"172.21.15.39:6829/3010986736","heartbeat_back_addr":"172.21.15.39:6833/3010986736","heartbeat_front_addr":"172.21.15.39:6831/3010986736","state":["exists","up"]},{"osd":10,"uuid":"9da2943f-b947-49d6-bab1-bc3a95299996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3982067769},{"type":"v1","addr":"172.21.15.101:6825","nonce":3982067769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":3982067769},{"type":"v1","addr":"172.21.15.101:6827","nonce":3982067769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":3982067769},{"type":"v1","addr":"172.21.15.101:6831","nonce":3982067769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":3982067769},{"type":"v1","addr":"172.21.15.101:6829","nonce":3982067769}]},"public_addr":"172.21.15.101:6825/3982067769","cluster_addr":"172.21.15.101:6827/3982067769","heartbeat_back_addr":"172.21.15.101:6831/3982067769","heartbeat_front_addr":"172.21.15.101:6829/3982067769","state":["exists","up"]},{"osd":11,"uuid":"ee5d51b4-c9ad-4603-a967-c60f3c1f2cdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":1265049047},{"type":"v1","addr":"172.21.15.133:6825","nonce":1265049047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6826","nonce":1265049047},{"type":"v1","addr":"172.21.15.133:6827","nonce":1265049047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6830","nonce":1265049047},{"type":"v1","addr":"172.21.15.133:6831","nonce":1265049047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6828","nonce":1265049047},{"type":"v1","addr":"172.21.15.133:6829","nonce":1265049047}]},"public_addr":"172.21.15.133:6825/1265049047","cluster_addr":"172.21.15.133:6827/1265049047","heartbeat_back_addr":"172.21.15.133:6831/1265049047","heartbeat_front_addr":"172.21.15.133:6829/1265049047","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:21:49.604087+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:22:11.298409+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:22:32.913915+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:22:56.095316+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:23:17.505593+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:23:38.490450+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:24:01.498630+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:24:22.952502+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:24:44.480122+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:25:07.141899+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:25:28.109970+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:25:49.227776+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:6801/1350590065":"2023-06-23T17:20:27.281966+0000","172.21.15.39:6800/1350590065":"2023-06-23T17:20:27.281966+0000","172.21.15.39:0/82302526":"2023-06-23T17:20:27.281966+0000","172.21.15.39:0/260892351":"2023-06-23T17:20:05.206927+0000","172.21.15.39:0/1513096987":"2023-06-23T17:20:05.206927+0000","172.21.15.39:6800/3960151496":"2023-06-23T17:20:05.206927+0000","172.21.15.39:6801/3960151496":"2023-06-23T17:20:05.206927+0000","172.21.15.39:0/2480502158":"2023-06-23T17:20:27.281966+0000","172.21.15.39:0/3848722689":"2023-06-23T17:20:27.281966+0000","172.21.15.39:0/2257210574":"2023-06-23T17:20:05.206927+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T17:26:01.114 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.112+0000 7f0932a7f700 1 -- 172.21.15.39:0/232191767 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f091c09b680 msgr2=0x7f091c09db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:01.114 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.112+0000 7f0932a7f700 1 --2- 172.21.15.39:0/232191767 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f091c09b680 0x7f091c09db30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0918004890 tx=0x7f0918009e30 comp rx=0 tx=0).stop 2023-06-22T17:26:01.115 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.112+0000 7f0932a7f700 1 -- 172.21.15.39:0/232191767 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f092c0fbbc0 msgr2=0x7f092c10c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:01.115 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.112+0000 7f0932a7f700 1 --2- 172.21.15.39:0/232191767 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f092c0fbbc0 0x7f092c10c9c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f092000b810 tx=0x7f0920009250 comp rx=0 tx=0).stop 2023-06-22T17:26:01.115 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.113+0000 7f0932a7f700 1 -- 172.21.15.39:0/232191767 shutdown_connections 2023-06-22T17:26:01.115 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.113+0000 7f0932a7f700 1 --2- 172.21.15.39:0/232191767 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f091c09b680 0x7f091c09db30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:01.115 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.113+0000 7f0932a7f700 1 --2- 172.21.15.39:0/232191767 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f092c0fd820 0x7f092c113f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:01.115 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.113+0000 7f0932a7f700 1 --2- 172.21.15.39:0/232191767 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f092c0fbbc0 0x7f092c10c9c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:01.116 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.113+0000 7f0932a7f700 1 --2- 172.21.15.39:0/232191767 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f092c0fce70 0x7f092c10cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:01.116 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.113+0000 7f0932a7f700 1 -- 172.21.15.39:0/232191767 >> 172.21.15.39:0/232191767 conn(0x7f092c0f7300 msgr2=0x7f092c0f94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:01.116 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.113+0000 7f0932a7f700 1 -- 172.21.15.39:0/232191767 shutdown_connections 2023-06-22T17:26:01.116 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.113+0000 7f0932a7f700 1 -- 172.21.15.39:0/232191767 wait complete. 2023-06-22T17:26:01.127 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-22T17:22:35.797853+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-06-22T17:26:01.128 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-22T17:26:01.289 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.287+0000 7f83f2aab700 1 -- 172.21.15.39:0/1688295135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83ec0fdf40 msgr2=0x7f83ec0fe390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:01.289 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.287+0000 7f83f2aab700 1 --2- 172.21.15.39:0/1688295135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83ec0fdf40 0x7f83ec0fe390 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f83d80049c0 tx=0x7f83d8009e30 comp rx=0 tx=0).stop 2023-06-22T17:26:01.290 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.287+0000 7f83f2aab700 1 -- 172.21.15.39:0/1688295135 shutdown_connections 2023-06-22T17:26:01.290 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.287+0000 7f83f2aab700 1 --2- 172.21.15.39:0/1688295135 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f83ec0fe8d0 0x7f83ec105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:01.290 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.287+0000 7f83f2aab700 1 --2- 172.21.15.39:0/1688295135 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83ec0fcd30 0x7f83ec0fd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:01.290 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.287+0000 7f83f2aab700 1 --2- 172.21.15.39:0/1688295135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83ec0fdf40 0x7f83ec0fe390 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:01.290 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.287+0000 7f83f2aab700 1 -- 172.21.15.39:0/1688295135 >> 172.21.15.39:0/1688295135 conn(0x7f83ec0f8490 msgr2=0x7f83ec0fa8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:01.290 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.287+0000 7f83f2aab700 1 -- 172.21.15.39:0/1688295135 shutdown_connections 2023-06-22T17:26:01.290 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.287+0000 7f83f2aab700 1 -- 172.21.15.39:0/1688295135 wait complete. 2023-06-22T17:26:01.291 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.287+0000 7f83f2aab700 1 Processor -- start 2023-06-22T17:26:01.291 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.288+0000 7f83f2aab700 1 -- start start 2023-06-22T17:26:01.292 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.288+0000 7f83f2aab700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f83ec0fcd30 0x7f83ec10d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:01.293 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.288+0000 7f83f2aab700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83ec0fdf40 0x7f83ec10de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:01.293 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.288+0000 7f83f2aab700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83ec0fe8d0 0x7f83ec114f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:01.293 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.288+0000 7f83f2aab700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ec115510 con 0x7f83ec0fdf40 2023-06-22T17:26:01.293 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.288+0000 7f83f2aab700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ec115680 con 0x7f83ec0fcd30 2023-06-22T17:26:01.293 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.288+0000 7f83f2aab700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ec115840 con 0x7f83ec0fe8d0 2023-06-22T17:26:01.293 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.288+0000 7f83ebfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83ec0fdf40 0x7f83ec10de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:01.293 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.288+0000 7f83ebfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83ec0fdf40 0x7f83ec10de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48928/0 (socket says 172.21.15.39:48928) 2023-06-22T17:26:01.294 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.288+0000 7f83ebfff700 1 -- 172.21.15.39:0/279867754 learned_addr learned my addr 172.21.15.39:0/279867754 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:26:01.294 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.288+0000 7f83f0847700 1 --2- 172.21.15.39:0/279867754 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f83ec0fcd30 0x7f83ec10d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:01.294 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.288+0000 7f83f1048700 1 --2- 172.21.15.39:0/279867754 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83ec0fe8d0 0x7f83ec114f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:01.294 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.288+0000 7f83ebfff700 1 -- 172.21.15.39:0/279867754 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83ec0fe8d0 msgr2=0x7f83ec114f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:01.294 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.288+0000 7f83ebfff700 1 --2- 172.21.15.39:0/279867754 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83ec0fe8d0 0x7f83ec114f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:01.294 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.288+0000 7f83ebfff700 1 -- 172.21.15.39:0/279867754 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f83ec0fcd30 msgr2=0x7f83ec10d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:01.294 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.288+0000 7f83ebfff700 1 --2- 172.21.15.39:0/279867754 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f83ec0fcd30 0x7f83ec10d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:01.295 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.288+0000 7f83ebfff700 1 -- 172.21.15.39:0/279867754 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d8005040 con 0x7f83ec0fdf40 2023-06-22T17:26:01.295 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.288+0000 7f83ebfff700 1 --2- 172.21.15.39:0/279867754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83ec0fdf40 0x7f83ec10de70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f83d8000cf0 tx=0x7f83d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:01.295 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.288+0000 7f83e97fa700 1 -- 172.21.15.39:0/279867754 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d800e040 con 0x7f83ec0fdf40 2023-06-22T17:26:01.295 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.289+0000 7f83e97fa700 1 -- 172.21.15.39:0/279867754 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f83d8009780 con 0x7f83ec0fdf40 2023-06-22T17:26:01.295 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.289+0000 7f83e97fa700 1 -- 172.21.15.39:0/279867754 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d801c3f0 con 0x7f83ec0fdf40 2023-06-22T17:26:01.295 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.289+0000 7f83f2aab700 1 -- 172.21.15.39:0/279867754 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83ec115ac0 con 0x7f83ec0fdf40 2023-06-22T17:26:01.295 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.289+0000 7f83f2aab700 1 -- 172.21.15.39:0/279867754 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83ec115f50 con 0x7f83ec0fdf40 2023-06-22T17:26:01.295 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.290+0000 7f83f2aab700 1 -- 172.21.15.39:0/279867754 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83ec1039c0 con 0x7f83ec0fdf40 2023-06-22T17:26:01.296 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.290+0000 7f83e97fa700 1 -- 172.21.15.39:0/279867754 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f83d80092a0 con 0x7f83ec0fdf40 2023-06-22T17:26:01.296 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.291+0000 7f83e97fa700 1 --2- 172.21.15.39:0/279867754 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f83dc09b730 0x7f83dc09dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:01.296 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.291+0000 7f83f0847700 1 --2- 172.21.15.39:0/279867754 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f83dc09b730 0x7f83dc09dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:01.296 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.291+0000 7f83e97fa700 1 -- 172.21.15.39:0/279867754 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f83d80c5a70 con 0x7f83ec0fdf40 2023-06-22T17:26:01.297 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.291+0000 7f83f0847700 1 --2- 172.21.15.39:0/279867754 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f83dc09b730 0x7f83dc09dbe0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f83ec0fdda0 tx=0x7f83e000b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:01.297 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.295+0000 7f83e97fa700 1 -- 172.21.15.39:0/279867754 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f83d8091770 con 0x7f83ec0fdf40 2023-06-22T17:26:01.491 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.489+0000 7f83f2aab700 1 -- 172.21.15.39:0/279867754 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f83ec1087f0 con 0x7f83ec0fdf40 2023-06-22T17:26:01.491 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.489+0000 7f83e97fa700 1 -- 172.21.15.39:0/279867754 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7f83d8094d90 con 0x7f83ec0fdf40 2023-06-22T17:26:01.491 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2023-06-22T17:26:01.493 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.491+0000 7f83f2aab700 1 -- 172.21.15.39:0/279867754 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f83dc09b730 msgr2=0x7f83dc09dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:01.494 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.491+0000 7f83f2aab700 1 --2- 172.21.15.39:0/279867754 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f83dc09b730 0x7f83dc09dbe0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f83ec0fdda0 tx=0x7f83e000b040 comp rx=0 tx=0).stop 2023-06-22T17:26:01.494 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.491+0000 7f83f2aab700 1 -- 172.21.15.39:0/279867754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83ec0fdf40 msgr2=0x7f83ec10de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:01.494 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.491+0000 7f83f2aab700 1 --2- 172.21.15.39:0/279867754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83ec0fdf40 0x7f83ec10de70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f83d8000cf0 tx=0x7f83d8009e30 comp rx=0 tx=0).stop 2023-06-22T17:26:01.494 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.491+0000 7f83f2aab700 1 -- 172.21.15.39:0/279867754 shutdown_connections 2023-06-22T17:26:01.494 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.491+0000 7f83f2aab700 1 --2- 172.21.15.39:0/279867754 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f83dc09b730 0x7f83dc09dbe0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:01.494 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.491+0000 7f83f2aab700 1 --2- 172.21.15.39:0/279867754 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83ec0fe8d0 0x7f83ec114f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:01.494 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.491+0000 7f83f2aab700 1 --2- 172.21.15.39:0/279867754 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f83ec0fcd30 0x7f83ec10d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:01.495 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.491+0000 7f83f2aab700 1 --2- 172.21.15.39:0/279867754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83ec0fdf40 0x7f83ec10de70 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:01.495 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.491+0000 7f83f2aab700 1 -- 172.21.15.39:0/279867754 >> 172.21.15.39:0/279867754 conn(0x7f83ec0f8490 msgr2=0x7f83ec0fa450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:01.495 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.491+0000 7f83f2aab700 1 -- 172.21.15.39:0/279867754 shutdown_connections 2023-06-22T17:26:01.495 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.491+0000 7f83f2aab700 1 -- 172.21.15.39:0/279867754 wait complete. 2023-06-22T17:26:01.506 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T17:26:01.617 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:01 smithi133 ceph-mon[86268]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi039=a;smithi039=d;smithi039=g;smithi039=j;smithi101=b;smithi101=e;smithi101=h;smithi101=k;smithi133=c;smithi133=f;smithi133=i;smithi133=l", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:26:01.618 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:01 smithi133 ceph-mon[86268]: Saving service mds.all spec with placement smithi039=a;smithi039=d;smithi039=g;smithi039=j;smithi101=b;smithi101=e;smithi101=h;smithi101=k;smithi133=c;smithi133=f;smithi133=i;smithi133=l;count:12 2023-06-22T17:26:01.618 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:01 smithi133 ceph-mon[86268]: Deploying daemon mds.l on smithi133 2023-06-22T17:26:01.618 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:01 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/232191767' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-06-22T17:26:01.618 INFO:journalctl@ceph.mds.l.smithi133.stdout:Jun 22 17:26:01 smithi133 systemd[1]: Starting Ceph mds.l for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:26:01.625 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.623+0000 7f05274ca700 1 -- 172.21.15.39:0/3229210743 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05200fdd70 msgr2=0x7f05200fe1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:01.626 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.623+0000 7f05274ca700 1 --2- 172.21.15.39:0/3229210743 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05200fdd70 0x7f05200fe1e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f05100049c0 tx=0x7f0510009e30 comp rx=0 tx=0).stop 2023-06-22T17:26:01.626 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.623+0000 7f05274ca700 1 -- 172.21.15.39:0/3229210743 shutdown_connections 2023-06-22T17:26:01.626 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.623+0000 7f05274ca700 1 --2- 172.21.15.39:0/3229210743 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f05200fe720 0x7f0520104fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:01.626 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.623+0000 7f05274ca700 1 --2- 172.21.15.39:0/3229210743 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05200fcac0 0x7f05200fced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:01.626 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.623+0000 7f05274ca700 1 --2- 172.21.15.39:0/3229210743 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05200fdd70 0x7f05200fe1e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:01.626 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.623+0000 7f05274ca700 1 -- 172.21.15.39:0/3229210743 >> 172.21.15.39:0/3229210743 conn(0x7f05200f8240 msgr2=0x7f05200fa650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:01.626 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.624+0000 7f05274ca700 1 -- 172.21.15.39:0/3229210743 shutdown_connections 2023-06-22T17:26:01.627 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.624+0000 7f05274ca700 1 -- 172.21.15.39:0/3229210743 wait complete. 2023-06-22T17:26:01.628 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.624+0000 7f05274ca700 1 Processor -- start 2023-06-22T17:26:01.628 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.624+0000 7f05274ca700 1 -- start start 2023-06-22T17:26:01.628 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.624+0000 7f05274ca700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05200fcac0 0x7f052010d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:01.629 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.624+0000 7f05274ca700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f05200fdd70 0x7f052010dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:01.629 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.624+0000 7f05274ca700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05200fe720 0x7f0520114d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:01.629 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.624+0000 7f05274ca700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0520115350 con 0x7f05200fcac0 2023-06-22T17:26:01.629 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.624+0000 7f05274ca700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05201154c0 con 0x7f05200fdd70 2023-06-22T17:26:01.629 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.624+0000 7f05274ca700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0520115680 con 0x7f05200fe720 2023-06-22T17:26:01.629 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.624+0000 7f05264c8700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05200fcac0 0x7f052010d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:01.629 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.624+0000 7f05264c8700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05200fcac0 0x7f052010d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48958/0 (socket says 172.21.15.39:48958) 2023-06-22T17:26:01.630 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.624+0000 7f05264c8700 1 -- 172.21.15.39:0/3952309885 learned_addr learned my addr 172.21.15.39:0/3952309885 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:26:01.630 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.624+0000 7f0526cc9700 1 --2- 172.21.15.39:0/3952309885 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05200fe720 0x7f0520114d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:01.630 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.624+0000 7f0525cc7700 1 --2- 172.21.15.39:0/3952309885 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f05200fdd70 0x7f052010dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:01.630 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.625+0000 7f05264c8700 1 -- 172.21.15.39:0/3952309885 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05200fe720 msgr2=0x7f0520114d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:01.630 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.625+0000 7f05264c8700 1 --2- 172.21.15.39:0/3952309885 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05200fe720 0x7f0520114d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:01.630 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.625+0000 7f05264c8700 1 -- 172.21.15.39:0/3952309885 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f05200fdd70 msgr2=0x7f052010dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:01.630 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.625+0000 7f05264c8700 1 --2- 172.21.15.39:0/3952309885 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f05200fdd70 0x7f052010dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:01.630 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.625+0000 7f05264c8700 1 -- 172.21.15.39:0/3952309885 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0510005040 con 0x7f05200fcac0 2023-06-22T17:26:01.631 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.625+0000 7f05264c8700 1 --2- 172.21.15.39:0/3952309885 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05200fcac0 0x7f052010d770 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f051c002ad0 tx=0x7f051c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:01.631 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.625+0000 7f0516ffd700 1 -- 172.21.15.39:0/3952309885 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051c007cc0 con 0x7f05200fcac0 2023-06-22T17:26:01.631 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.625+0000 7f05274ca700 1 -- 172.21.15.39:0/3952309885 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0520115960 con 0x7f05200fcac0 2023-06-22T17:26:01.631 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.625+0000 7f0516ffd700 1 -- 172.21.15.39:0/3952309885 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f051c00f040 con 0x7f05200fcac0 2023-06-22T17:26:01.631 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.625+0000 7f0516ffd700 1 -- 172.21.15.39:0/3952309885 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051c00dae0 con 0x7f05200fcac0 2023-06-22T17:26:01.631 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.625+0000 7f05274ca700 1 -- 172.21.15.39:0/3952309885 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0520115ed0 con 0x7f05200fcac0 2023-06-22T17:26:01.631 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.626+0000 7f05274ca700 1 -- 172.21.15.39:0/3952309885 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0520103f90 con 0x7f05200fcac0 2023-06-22T17:26:01.632 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.627+0000 7f0516ffd700 1 -- 172.21.15.39:0/3952309885 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f051c0090d0 con 0x7f05200fcac0 2023-06-22T17:26:01.632 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.628+0000 7f0516ffd700 1 --2- 172.21.15.39:0/3952309885 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f050c09b680 0x7f050c09db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:01.632 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.628+0000 7f0516ffd700 1 -- 172.21.15.39:0/3952309885 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f051c0c5110 con 0x7f05200fcac0 2023-06-22T17:26:01.633 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.628+0000 7f0525cc7700 1 --2- 172.21.15.39:0/3952309885 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f050c09b680 0x7f050c09db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:01.633 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.628+0000 7f0525cc7700 1 --2- 172.21.15.39:0/3952309885 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f050c09b680 0x7f050c09db30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f051000b3b0 tx=0x7f0510009e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:01.633 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.631+0000 7f0516ffd700 1 -- 172.21.15.39:0/3952309885 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f051c090e10 con 0x7f05200fcac0 2023-06-22T17:26:01.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:01 smithi101 ceph-mon[85507]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi039=a;smithi039=d;smithi039=g;smithi039=j;smithi101=b;smithi101=e;smithi101=h;smithi101=k;smithi133=c;smithi133=f;smithi133=i;smithi133=l", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:26:01.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:01 smithi101 ceph-mon[85507]: Saving service mds.all spec with placement smithi039=a;smithi039=d;smithi039=g;smithi039=j;smithi101=b;smithi101=e;smithi101=h;smithi101=k;smithi133=c;smithi133=f;smithi133=i;smithi133=l;count:12 2023-06-22T17:26:01.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:01 smithi101 ceph-mon[85507]: Deploying daemon mds.l on smithi133 2023-06-22T17:26:01.878 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:01 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/232191767' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-06-22T17:26:01.878 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.876+0000 7f05274ca700 1 -- 172.21.15.39:0/3952309885 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0520108410 con 0x7f05200fcac0 2023-06-22T17:26:01.879 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.876+0000 7f0516ffd700 1 -- 172.21.15.39:0/3952309885 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7f051c094430 con 0x7f05200fcac0 2023-06-22T17:26:01.879 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:26:01.879 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2023-06-22T17:26:01.881 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.878+0000 7f05274ca700 1 -- 172.21.15.39:0/3952309885 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f050c09b680 msgr2=0x7f050c09db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:01.881 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.878+0000 7f05274ca700 1 --2- 172.21.15.39:0/3952309885 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f050c09b680 0x7f050c09db30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f051000b3b0 tx=0x7f0510009e30 comp rx=0 tx=0).stop 2023-06-22T17:26:01.881 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.878+0000 7f05274ca700 1 -- 172.21.15.39:0/3952309885 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05200fcac0 msgr2=0x7f052010d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:01.881 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.878+0000 7f05274ca700 1 --2- 172.21.15.39:0/3952309885 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05200fcac0 0x7f052010d770 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f051c002ad0 tx=0x7f051c0046b0 comp rx=0 tx=0).stop 2023-06-22T17:26:01.881 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.879+0000 7f05274ca700 1 -- 172.21.15.39:0/3952309885 shutdown_connections 2023-06-22T17:26:01.881 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.879+0000 7f05274ca700 1 --2- 172.21.15.39:0/3952309885 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f050c09b680 0x7f050c09db30 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:01.882 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.879+0000 7f05274ca700 1 --2- 172.21.15.39:0/3952309885 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05200fe720 0x7f0520114d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:01.882 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.879+0000 7f05274ca700 1 --2- 172.21.15.39:0/3952309885 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05200fcac0 0x7f052010d770 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:01.882 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.879+0000 7f05274ca700 1 --2- 172.21.15.39:0/3952309885 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f05200fdd70 0x7f052010dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:01.882 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.879+0000 7f05274ca700 1 -- 172.21.15.39:0/3952309885 >> 172.21.15.39:0/3952309885 conn(0x7f05200f8240 msgr2=0x7f0520101110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:01.882 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.879+0000 7f05274ca700 1 -- 172.21.15.39:0/3952309885 shutdown_connections 2023-06-22T17:26:01.882 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:01.879+0000 7f05274ca700 1 -- 172.21.15.39:0/3952309885 wait complete. 2023-06-22T17:26:01.883 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 1 2023-06-22T17:26:01.891 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:01 smithi039 ceph-mon[84805]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi039=a;smithi039=d;smithi039=g;smithi039=j;smithi101=b;smithi101=e;smithi101=h;smithi101=k;smithi133=c;smithi133=f;smithi133=i;smithi133=l", "target": ["mon-mgr", ""]}]: dispatch 2023-06-22T17:26:01.892 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:01 smithi039 ceph-mon[84805]: Saving service mds.all spec with placement smithi039=a;smithi039=d;smithi039=g;smithi039=j;smithi101=b;smithi101=e;smithi101=h;smithi101=k;smithi133=c;smithi133=f;smithi133=i;smithi133=l;count:12 2023-06-22T17:26:01.892 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:01 smithi039 ceph-mon[84805]: Deploying daemon mds.l on smithi133 2023-06-22T17:26:01.892 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:01 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/232191767' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-06-22T17:26:02.421 INFO:journalctl@ceph.mds.l.smithi133.stdout:Jun 22 17:26:02 smithi133 bash[118846]: 8786ea157bec9a8904ed8029afd0de871f7f6e3ec6e75ae08f09342a0261d2ef 2023-06-22T17:26:02.421 INFO:journalctl@ceph.mds.l.smithi133.stdout:Jun 22 17:26:02 smithi133 ceph-mds[118910]: -- 172.21.15.133:0/2988474842 <== mon.1 v2:172.21.15.133:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558fa1c669c0 con 0x558fa1c3cc00 2023-06-22T17:26:02.421 INFO:journalctl@ceph.mds.l.smithi133.stdout:Jun 22 17:26:02 smithi133 conmon[118886]: starting mds.l at 2023-06-22T17:26:02.421 INFO:journalctl@ceph.mds.l.smithi133.stdout:Jun 22 17:26:02 smithi133 systemd[1]: Started Ceph mds.l for d88002e2-1120-11ee-9b2b-001a4aab830c. 2023-06-22T17:26:02.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:02 smithi101 ceph-mon[85507]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:02.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:02 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/279867754' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-06-22T17:26:02.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:02 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/3952309885' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:02.878 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:02 smithi101 ceph-mon[85507]: mds.? [v2:172.21.15.133:6832/1691506923,v1:172.21.15.133:6833/1691506923] up:boot 2023-06-22T17:26:02.878 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:02 smithi101 ceph-mon[85507]: fsmap 1 up:standby 2023-06-22T17:26:02.878 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:02 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-06-22T17:26:02.879 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:02 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:02.879 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:02 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:02.879 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:02 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:02.879 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:02 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:02.879 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:02 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:02.879 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:02 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:02.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:02 smithi133 ceph-mon[86268]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:02.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:02 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/279867754' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-06-22T17:26:02.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:02 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/3952309885' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:02.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:02 smithi133 ceph-mon[86268]: mds.? [v2:172.21.15.133:6832/1691506923,v1:172.21.15.133:6833/1691506923] up:boot 2023-06-22T17:26:02.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:02 smithi133 ceph-mon[86268]: fsmap 1 up:standby 2023-06-22T17:26:02.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-06-22T17:26:02.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:02.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:02.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:02.923 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:02.923 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:02.923 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:02 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:02.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:02 smithi039 ceph-mon[84805]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:02.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:02 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/279867754' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-06-22T17:26:02.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:02 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/3952309885' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:02.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:02 smithi039 ceph-mon[84805]: mds.? [v2:172.21.15.133:6832/1691506923,v1:172.21.15.133:6833/1691506923] up:boot 2023-06-22T17:26:02.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:02 smithi039 ceph-mon[84805]: fsmap 1 up:standby 2023-06-22T17:26:02.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:02 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-06-22T17:26:02.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:02 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:02.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:02 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:02.935 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:02 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:02.935 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:02 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:02.935 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:02 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:02.935 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:02 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:03.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:03 smithi101 ceph-mon[85507]: Deploying daemon mds.k on smithi101 2023-06-22T17:26:03.894 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T17:26:03.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:03 smithi133 ceph-mon[86268]: Deploying daemon mds.k on smithi101 2023-06-22T17:26:03.921 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:03 smithi039 ceph-mon[84805]: Deploying daemon mds.k on smithi101 2023-06-22T17:26:04.028 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.026+0000 7fda35220700 1 -- 172.21.15.39:0/1322999187 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda30100130 msgr2=0x7fda30102510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:04.028 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.026+0000 7fda35220700 1 --2- 172.21.15.39:0/1322999187 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda30100130 0x7fda30102510 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fda180049c0 tx=0x7fda18009e30 comp rx=0 tx=0).stop 2023-06-22T17:26:04.028 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.026+0000 7fda35220700 1 -- 172.21.15.39:0/1322999187 shutdown_connections 2023-06-22T17:26:04.028 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.026+0000 7fda35220700 1 --2- 172.21.15.39:0/1322999187 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fda30102a50 0x7fda30104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:04.029 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.026+0000 7fda35220700 1 --2- 172.21.15.39:0/1322999187 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fda300fd810 0x7fda300ffbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:04.029 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.026+0000 7fda35220700 1 --2- 172.21.15.39:0/1322999187 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda30100130 0x7fda30102510 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:04.029 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.026+0000 7fda35220700 1 -- 172.21.15.39:0/1322999187 >> 172.21.15.39:0/1322999187 conn(0x7fda300f7310 msgr2=0x7fda300f9760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:04.029 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.026+0000 7fda35220700 1 -- 172.21.15.39:0/1322999187 shutdown_connections 2023-06-22T17:26:04.029 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.026+0000 7fda35220700 1 -- 172.21.15.39:0/1322999187 wait complete. 2023-06-22T17:26:04.031 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.026+0000 7fda35220700 1 Processor -- start 2023-06-22T17:26:04.031 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.027+0000 7fda35220700 1 -- start start 2023-06-22T17:26:04.032 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.027+0000 7fda35220700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fda300fd810 0x7fda3010dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:04.032 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.027+0000 7fda35220700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fda30100130 0x7fda3010e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:04.032 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.027+0000 7fda35220700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda30102a50 0x7fda301150b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:04.032 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.027+0000 7fda35220700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda301156c0 con 0x7fda30102a50 2023-06-22T17:26:04.032 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.027+0000 7fda35220700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda30115830 con 0x7fda30100130 2023-06-22T17:26:04.032 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.027+0000 7fda35220700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda301159f0 con 0x7fda300fd810 2023-06-22T17:26:04.032 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.027+0000 7fda2f59e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda30102a50 0x7fda301150b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:04.033 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.027+0000 7fda2f59e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda30102a50 0x7fda301150b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48968/0 (socket says 172.21.15.39:48968) 2023-06-22T17:26:04.033 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.027+0000 7fda2f59e700 1 -- 172.21.15.39:0/287833317 learned_addr learned my addr 172.21.15.39:0/287833317 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:26:04.033 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.027+0000 7fda2ed9d700 1 --2- 172.21.15.39:0/287833317 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fda300fd810 0x7fda3010dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:04.033 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.027+0000 7fda2e59c700 1 --2- 172.21.15.39:0/287833317 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fda30100130 0x7fda3010e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:04.033 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.027+0000 7fda2f59e700 1 -- 172.21.15.39:0/287833317 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fda300fd810 msgr2=0x7fda3010dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:04.033 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.027+0000 7fda2f59e700 1 --2- 172.21.15.39:0/287833317 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fda300fd810 0x7fda3010dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:04.033 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.027+0000 7fda2f59e700 1 -- 172.21.15.39:0/287833317 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fda30100130 msgr2=0x7fda3010e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:04.034 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.027+0000 7fda2f59e700 1 --2- 172.21.15.39:0/287833317 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fda30100130 0x7fda3010e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:04.034 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.027+0000 7fda2f59e700 1 -- 172.21.15.39:0/287833317 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda18005040 con 0x7fda30102a50 2023-06-22T17:26:04.034 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.027+0000 7fda2f59e700 1 --2- 172.21.15.39:0/287833317 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda30102a50 0x7fda301150b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fda2800c7f0 tx=0x7fda28004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:04.034 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.028+0000 7fda277fe700 1 -- 172.21.15.39:0/287833317 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda28012630 con 0x7fda30102a50 2023-06-22T17:26:04.034 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.028+0000 7fda35220700 1 -- 172.21.15.39:0/287833317 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda30115cd0 con 0x7fda30102a50 2023-06-22T17:26:04.034 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.028+0000 7fda277fe700 1 -- 172.21.15.39:0/287833317 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7fda28012c70 con 0x7fda30102a50 2023-06-22T17:26:04.034 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.028+0000 7fda277fe700 1 -- 172.21.15.39:0/287833317 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda28011cb0 con 0x7fda30102a50 2023-06-22T17:26:04.034 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.028+0000 7fda35220700 1 -- 172.21.15.39:0/287833317 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda30116170 con 0x7fda30102a50 2023-06-22T17:26:04.035 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.029+0000 7fda35220700 1 -- 172.21.15.39:0/287833317 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda300f8e40 con 0x7fda30102a50 2023-06-22T17:26:04.035 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.030+0000 7fda277fe700 1 -- 172.21.15.39:0/287833317 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7fda28012790 con 0x7fda30102a50 2023-06-22T17:26:04.035 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.031+0000 7fda277fe700 1 --2- 172.21.15.39:0/287833317 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fda1c09b680 0x7fda1c09db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:04.036 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.031+0000 7fda2ed9d700 1 --2- 172.21.15.39:0/287833317 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fda1c09b680 0x7fda1c09db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:04.036 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.031+0000 7fda277fe700 1 -- 172.21.15.39:0/287833317 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fda280c5270 con 0x7fda30102a50 2023-06-22T17:26:04.036 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.031+0000 7fda2ed9d700 1 --2- 172.21.15.39:0/287833317 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fda1c09b680 0x7fda1c09db30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fda20007580 tx=0x7fda20009780 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:04.036 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.034+0000 7fda277fe700 1 -- 172.21.15.39:0/287833317 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fda28090ed0 con 0x7fda30102a50 2023-06-22T17:26:04.126 INFO:journalctl@ceph.mds.k.smithi101.stdout:Jun 22 17:26:03 smithi101 systemd[1]: Starting Ceph mds.k for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:26:04.276 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.274+0000 7fda35220700 1 -- 172.21.15.39:0/287833317 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda30108670 con 0x7fda30102a50 2023-06-22T17:26:04.276 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.274+0000 7fda277fe700 1 -- 172.21.15.39:0/287833317 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1260 (secure 0 0 0) 0x7fda28016090 con 0x7fda30102a50 2023-06-22T17:26:04.277 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:26:04.277 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/1691506923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":1691506923},{"type":"v1","addr":"172.21.15.133:6833","nonce":1691506923}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-06-22T17:26:04.278 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.276+0000 7fda35220700 1 -- 172.21.15.39:0/287833317 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fda1c09b680 msgr2=0x7fda1c09db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:04.278 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.276+0000 7fda35220700 1 --2- 172.21.15.39:0/287833317 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fda1c09b680 0x7fda1c09db30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fda20007580 tx=0x7fda20009780 comp rx=0 tx=0).stop 2023-06-22T17:26:04.278 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.276+0000 7fda35220700 1 -- 172.21.15.39:0/287833317 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda30102a50 msgr2=0x7fda301150b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:04.279 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.276+0000 7fda35220700 1 --2- 172.21.15.39:0/287833317 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda30102a50 0x7fda301150b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fda2800c7f0 tx=0x7fda28004470 comp rx=0 tx=0).stop 2023-06-22T17:26:04.279 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.276+0000 7fda35220700 1 -- 172.21.15.39:0/287833317 shutdown_connections 2023-06-22T17:26:04.279 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.276+0000 7fda35220700 1 --2- 172.21.15.39:0/287833317 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fda1c09b680 0x7fda1c09db30 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:04.279 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.276+0000 7fda35220700 1 --2- 172.21.15.39:0/287833317 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda30102a50 0x7fda301150b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:04.279 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.276+0000 7fda35220700 1 --2- 172.21.15.39:0/287833317 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fda300fd810 0x7fda3010dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:04.279 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.276+0000 7fda35220700 1 --2- 172.21.15.39:0/287833317 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fda30100130 0x7fda3010e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:04.279 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.276+0000 7fda35220700 1 -- 172.21.15.39:0/287833317 >> 172.21.15.39:0/287833317 conn(0x7fda300f7310 msgr2=0x7fda300fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:04.280 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.276+0000 7fda35220700 1 -- 172.21.15.39:0/287833317 shutdown_connections 2023-06-22T17:26:04.280 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:04.276+0000 7fda35220700 1 -- 172.21.15.39:0/287833317 wait complete. 2023-06-22T17:26:04.280 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 2 2023-06-22T17:26:04.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:04 smithi101 ceph-mon[85507]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:04.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:04 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/287833317' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:04.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:04 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:04.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:04 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:04.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:04 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:04.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:04 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:04.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:04 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:04.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:04 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:04.628 INFO:journalctl@ceph.mds.k.smithi101.stdout:Jun 22 17:26:04 smithi101 ceph-mds[116508]: -- 172.21.15.101:0/1134436177 <== mon.2 v2:172.21.15.101:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563fcda769c0 con 0x563fcda4cc00 2023-06-22T17:26:04.628 INFO:journalctl@ceph.mds.k.smithi101.stdout:Jun 22 17:26:04 smithi101 conmon[116484]: starting mds.k at 2023-06-22T17:26:04.628 INFO:journalctl@ceph.mds.k.smithi101.stdout:Jun 22 17:26:04 smithi101 bash[116444]: 1ef1584cc3bb57acf843e0561effc886fce17eeb23c7d95729655c1b2e2fee59 2023-06-22T17:26:04.628 INFO:journalctl@ceph.mds.k.smithi101.stdout:Jun 22 17:26:04 smithi101 systemd[1]: Started Ceph mds.k for d88002e2-1120-11ee-9b2b-001a4aab830c. 2023-06-22T17:26:04.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:04 smithi039 ceph-mon[84805]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:04.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:04 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/287833317' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:04.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:04 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:04.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:04 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:04.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:04 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:04.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:04 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:04.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:04 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:04.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:04 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:04.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:04 smithi133 ceph-mon[86268]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:04.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:04 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/287833317' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:04.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:04 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:04.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:04 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:04.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:04 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:04.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:04 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:04.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:04 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:04.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:04 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:05.775 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:05 smithi039 ceph-mon[84805]: Deploying daemon mds.j on smithi039 2023-06-22T17:26:05.775 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:05 smithi039 ceph-mon[84805]: mds.? [v2:172.21.15.101:6832/3120964256,v1:172.21.15.101:6833/3120964256] up:boot 2023-06-22T17:26:05.776 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:05 smithi039 ceph-mon[84805]: fsmap 2 up:standby 2023-06-22T17:26:05.776 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:05 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-06-22T17:26:05.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:05 smithi101 ceph-mon[85507]: Deploying daemon mds.j on smithi039 2023-06-22T17:26:05.878 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:05 smithi101 ceph-mon[85507]: mds.? [v2:172.21.15.101:6832/3120964256,v1:172.21.15.101:6833/3120964256] up:boot 2023-06-22T17:26:05.878 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:05 smithi101 ceph-mon[85507]: fsmap 2 up:standby 2023-06-22T17:26:05.878 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:05 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-06-22T17:26:05.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:05 smithi133 ceph-mon[86268]: Deploying daemon mds.j on smithi039 2023-06-22T17:26:05.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:05 smithi133 ceph-mon[86268]: mds.? [v2:172.21.15.101:6832/3120964256,v1:172.21.15.101:6833/3120964256] up:boot 2023-06-22T17:26:05.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:05 smithi133 ceph-mon[86268]: fsmap 2 up:standby 2023-06-22T17:26:05.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:05 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-06-22T17:26:06.184 INFO:journalctl@ceph.mds.j.smithi039.stdout:Jun 22 17:26:05 smithi039 systemd[1]: Starting Ceph mds.j for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:26:06.291 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T17:26:06.422 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.420+0000 7f57f9805700 1 -- 172.21.15.39:0/1733143862 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57f40fbbd0 msgr2=0x7f57f40fbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:06.422 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.420+0000 7f57f9805700 1 --2- 172.21.15.39:0/1733143862 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57f40fbbd0 0x7f57f40fbfe0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f57e4009d60 tx=0x7f57e4004d10 comp rx=0 tx=0).stop 2023-06-22T17:26:06.422 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.420+0000 7f57f9805700 1 -- 172.21.15.39:0/1733143862 shutdown_connections 2023-06-22T17:26:06.422 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.420+0000 7f57f9805700 1 --2- 172.21.15.39:0/1733143862 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f57f40fd770 0x7f57f4104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:06.422 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.420+0000 7f57f9805700 1 --2- 172.21.15.39:0/1733143862 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57f40fbbd0 0x7f57f40fbfe0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:06.423 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.420+0000 7f57f9805700 1 --2- 172.21.15.39:0/1733143862 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57f40fcde0 0x7f57f40fd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:06.423 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.420+0000 7f57f9805700 1 -- 172.21.15.39:0/1733143862 >> 172.21.15.39:0/1733143862 conn(0x7f57f40f7310 msgr2=0x7f57f40f9760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:06.423 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.420+0000 7f57f9805700 1 -- 172.21.15.39:0/1733143862 shutdown_connections 2023-06-22T17:26:06.423 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.420+0000 7f57f9805700 1 -- 172.21.15.39:0/1733143862 wait complete. 2023-06-22T17:26:06.425 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.421+0000 7f57f9805700 1 Processor -- start 2023-06-22T17:26:06.425 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.421+0000 7f57f9805700 1 -- start start 2023-06-22T17:26:06.425 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.421+0000 7f57f9805700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57f40fbbd0 0x7f57f4065a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:06.425 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.421+0000 7f57f9805700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f57f40fcde0 0x7f57f4112850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:06.425 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.421+0000 7f57f9805700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57f40fd770 0x7f57f4114e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:06.425 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.421+0000 7f57f9805700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f4115630 con 0x7f57f40fd770 2023-06-22T17:26:06.426 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.421+0000 7f57f9805700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f41157a0 con 0x7f57f40fbbd0 2023-06-22T17:26:06.426 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.421+0000 7f57f9805700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f4115960 con 0x7f57f40fcde0 2023-06-22T17:26:06.426 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.421+0000 7f57f37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57f40fd770 0x7f57f4114e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:06.426 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.421+0000 7f57f2ffd700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57f40fbbd0 0x7f57f4065a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:06.426 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.421+0000 7f57f37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57f40fd770 0x7f57f4114e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49298/0 (socket says 172.21.15.39:49298) 2023-06-22T17:26:06.426 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.421+0000 7f57f37fe700 1 -- 172.21.15.39:0/2126090937 learned_addr learned my addr 172.21.15.39:0/2126090937 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:26:06.427 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.421+0000 7f57f27fc700 1 --2- 172.21.15.39:0/2126090937 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f57f40fcde0 0x7f57f4112850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:06.427 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.421+0000 7f57f37fe700 1 -- 172.21.15.39:0/2126090937 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f57f40fcde0 msgr2=0x7f57f4112850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:06.427 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.421+0000 7f57f37fe700 1 --2- 172.21.15.39:0/2126090937 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f57f40fcde0 0x7f57f4112850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:06.427 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.421+0000 7f57f37fe700 1 -- 172.21.15.39:0/2126090937 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57f40fbbd0 msgr2=0x7f57f4065a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:06.427 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.421+0000 7f57f37fe700 1 --2- 172.21.15.39:0/2126090937 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57f40fbbd0 0x7f57f4065a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:06.427 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.421+0000 7f57f37fe700 1 -- 172.21.15.39:0/2126090937 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57dc005040 con 0x7f57f40fd770 2023-06-22T17:26:06.428 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.422+0000 7f57f37fe700 1 --2- 172.21.15.39:0/2126090937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57f40fd770 0x7f57f4114e90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f57ec00eb70 tx=0x7f57ec00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:06.428 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.422+0000 7f57eb7fe700 1 -- 172.21.15.39:0/2126090937 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ec007cc0 con 0x7f57f40fd770 2023-06-22T17:26:06.428 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.422+0000 7f57f9805700 1 -- 172.21.15.39:0/2126090937 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57e4009940 con 0x7f57f40fd770 2023-06-22T17:26:06.428 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.422+0000 7f57eb7fe700 1 -- 172.21.15.39:0/2126090937 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f57ec00f040 con 0x7f57f40fd770 2023-06-22T17:26:06.428 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.422+0000 7f57eb7fe700 1 -- 172.21.15.39:0/2126090937 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ec00c9c0 con 0x7f57f40fd770 2023-06-22T17:26:06.428 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.422+0000 7f57f9805700 1 -- 172.21.15.39:0/2126090937 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57f4115f70 con 0x7f57f40fd770 2023-06-22T17:26:06.428 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.423+0000 7f57f9805700 1 -- 172.21.15.39:0/2126090937 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57f4102ae0 con 0x7f57f40fd770 2023-06-22T17:26:06.429 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.424+0000 7f57eb7fe700 1 -- 172.21.15.39:0/2126090937 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f57ec0090d0 con 0x7f57f40fd770 2023-06-22T17:26:06.429 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.425+0000 7f57eb7fe700 1 --2- 172.21.15.39:0/2126090937 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f57e009b680 0x7f57e009db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:06.429 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.425+0000 7f57eb7fe700 1 -- 172.21.15.39:0/2126090937 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f57ec0c44c0 con 0x7f57f40fd770 2023-06-22T17:26:06.430 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.425+0000 7f57f2ffd700 1 --2- 172.21.15.39:0/2126090937 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f57e009b680 0x7f57e009db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:06.430 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.425+0000 7f57f2ffd700 1 --2- 172.21.15.39:0/2126090937 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f57e009b680 0x7f57e009db30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f57e4009d30 tx=0x7f57e4004d10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:06.431 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.429+0000 7f57eb7fe700 1 -- 172.21.15.39:0/2126090937 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f57ec04da00 con 0x7f57f40fd770 2023-06-22T17:26:06.683 INFO:journalctl@ceph.mds.j.smithi039.stdout:Jun 22 17:26:06 smithi039 ceph-mds[128075]: -- 172.21.15.39:0/3722858068 <== mon.0 v2:172.21.15.39:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b32ef689c0 con 0x55b32ef3ec00 2023-06-22T17:26:06.683 INFO:journalctl@ceph.mds.j.smithi039.stdout:Jun 22 17:26:06 smithi039 conmon[128031]: starting mds.j at 2023-06-22T17:26:06.684 INFO:journalctl@ceph.mds.j.smithi039.stdout:Jun 22 17:26:06 smithi039 bash[127972]: 80d0e0a3621a143bfedb70d31e55252543f71c4ca7ad674383f84caf8de3552f 2023-06-22T17:26:06.684 INFO:journalctl@ceph.mds.j.smithi039.stdout:Jun 22 17:26:06 smithi039 systemd[1]: Started Ceph mds.j for d88002e2-1120-11ee-9b2b-001a4aab830c. 2023-06-22T17:26:06.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:06 smithi039 ceph-mon[84805]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:06.709 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.705+0000 7f57f9805700 1 -- 172.21.15.39:0/2126090937 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57f4107500 con 0x7f57f40fd770 2023-06-22T17:26:06.709 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.705+0000 7f57eb7fe700 1 -- 172.21.15.39:0/2126090937 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2020 (secure 0 0 0) 0x7f57ec019800 con 0x7f57f40fd770 2023-06-22T17:26:06.710 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:26:06.710 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6833/3120964256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":3120964256},{"type":"v1","addr":"172.21.15.101:6833","nonce":3120964256}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/1691506923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":1691506923},{"type":"v1","addr":"172.21.15.133:6833","nonce":1691506923}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-06-22T17:26:06.711 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.707+0000 7f57f9805700 1 -- 172.21.15.39:0/2126090937 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f57e009b680 msgr2=0x7f57e009db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:06.711 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.707+0000 7f57f9805700 1 --2- 172.21.15.39:0/2126090937 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f57e009b680 0x7f57e009db30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f57e4009d30 tx=0x7f57e4004d10 comp rx=0 tx=0).stop 2023-06-22T17:26:06.711 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.707+0000 7f57f9805700 1 -- 172.21.15.39:0/2126090937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57f40fd770 msgr2=0x7f57f4114e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:06.711 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.707+0000 7f57f9805700 1 --2- 172.21.15.39:0/2126090937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57f40fd770 0x7f57f4114e90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f57ec00eb70 tx=0x7f57ec00b040 comp rx=0 tx=0).stop 2023-06-22T17:26:06.711 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.708+0000 7f57f9805700 1 -- 172.21.15.39:0/2126090937 shutdown_connections 2023-06-22T17:26:06.711 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.708+0000 7f57f9805700 1 --2- 172.21.15.39:0/2126090937 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f57e009b680 0x7f57e009db30 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:06.712 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.708+0000 7f57f9805700 1 --2- 172.21.15.39:0/2126090937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57f40fd770 0x7f57f4114e90 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:06.712 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.708+0000 7f57f9805700 1 --2- 172.21.15.39:0/2126090937 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57f40fbbd0 0x7f57f4065a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:06.712 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.708+0000 7f57f9805700 1 --2- 172.21.15.39:0/2126090937 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f57f40fcde0 0x7f57f4112850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:06.712 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.708+0000 7f57f9805700 1 -- 172.21.15.39:0/2126090937 >> 172.21.15.39:0/2126090937 conn(0x7f57f40f7310 msgr2=0x7f57f40f9590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:06.712 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.708+0000 7f57f9805700 1 -- 172.21.15.39:0/2126090937 shutdown_connections 2023-06-22T17:26:06.712 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:06.708+0000 7f57f9805700 1 -- 172.21.15.39:0/2126090937 wait complete. 2023-06-22T17:26:06.712 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 3 2023-06-22T17:26:06.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:06 smithi101 ceph-mon[85507]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:06.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:06 smithi133 ceph-mon[86268]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:07.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:07 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:07.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:07 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:07.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:07 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:07.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:07 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:07.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:07 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:07.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:07 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:07.878 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:07 smithi101 ceph-mon[85507]: Deploying daemon mds.d on smithi039 2023-06-22T17:26:07.878 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:07 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/2126090937' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:07.878 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:07 smithi101 ceph-mon[85507]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:07.878 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:07 smithi101 ceph-mon[85507]: mds.? [v2:172.21.15.39:6834/3630756954,v1:172.21.15.39:6835/3630756954] up:boot 2023-06-22T17:26:07.878 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:07 smithi101 ceph-mon[85507]: fsmap 3 up:standby 2023-06-22T17:26:07.878 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:07 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-06-22T17:26:07.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:07 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:07.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:07 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:07.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:07 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:07.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:07 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:07.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:07 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:07.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:07 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:07.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:07 smithi133 ceph-mon[86268]: Deploying daemon mds.d on smithi039 2023-06-22T17:26:07.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:07 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/2126090937' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:07.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:07 smithi133 ceph-mon[86268]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:07.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:07 smithi133 ceph-mon[86268]: mds.? [v2:172.21.15.39:6834/3630756954,v1:172.21.15.39:6835/3630756954] up:boot 2023-06-22T17:26:07.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:07 smithi133 ceph-mon[86268]: fsmap 3 up:standby 2023-06-22T17:26:07.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:07 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-06-22T17:26:07.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:07 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:07.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:07 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:07.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:07 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:07.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:07 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:07.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:07 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:07.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:07 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:07.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:07 smithi039 ceph-mon[84805]: Deploying daemon mds.d on smithi039 2023-06-22T17:26:07.935 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:07 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/2126090937' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:07.935 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:07 smithi039 ceph-mon[84805]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:07.935 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:07 smithi039 ceph-mon[84805]: mds.? [v2:172.21.15.39:6834/3630756954,v1:172.21.15.39:6835/3630756954] up:boot 2023-06-22T17:26:07.935 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:07 smithi039 ceph-mon[84805]: fsmap 3 up:standby 2023-06-22T17:26:07.935 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:07 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-06-22T17:26:08.434 INFO:journalctl@ceph.mds.d.smithi039.stdout:Jun 22 17:26:08 smithi039 systemd[1]: Starting Ceph mds.d for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:26:08.724 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T17:26:08.862 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.860+0000 7f6ded435700 1 -- 172.21.15.39:0/1724664805 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6de80fbbd0 msgr2=0x7f6de80fbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:08.863 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.860+0000 7f6ded435700 1 --2- 172.21.15.39:0/1724664805 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6de80fbbd0 0x7f6de80fbfe0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6dd00049c0 tx=0x7f6dd0009e30 comp rx=0 tx=0).stop 2023-06-22T17:26:08.863 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.860+0000 7f6ded435700 1 -- 172.21.15.39:0/1724664805 shutdown_connections 2023-06-22T17:26:08.863 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.860+0000 7f6ded435700 1 --2- 172.21.15.39:0/1724664805 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f6de80fd770 0x7f6de8104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:08.864 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.860+0000 7f6ded435700 1 --2- 172.21.15.39:0/1724664805 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6de80fbbd0 0x7f6de80fbfe0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:08.864 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.860+0000 7f6ded435700 1 --2- 172.21.15.39:0/1724664805 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6de80fcde0 0x7f6de80fd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:08.865 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.860+0000 7f6ded435700 1 -- 172.21.15.39:0/1724664805 >> 172.21.15.39:0/1724664805 conn(0x7f6de80f7310 msgr2=0x7f6de80f9760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:08.865 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.860+0000 7f6ded435700 1 -- 172.21.15.39:0/1724664805 shutdown_connections 2023-06-22T17:26:08.865 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.860+0000 7f6ded435700 1 -- 172.21.15.39:0/1724664805 wait complete. 2023-06-22T17:26:08.868 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.861+0000 7f6ded435700 1 Processor -- start 2023-06-22T17:26:08.868 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.861+0000 7f6ded435700 1 -- start start 2023-06-22T17:26:08.868 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.861+0000 7f6ded435700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6de80fbbd0 0x7f6de810fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:08.869 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.861+0000 7f6ded435700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6de80fcde0 0x7f6de8110170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:08.869 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.861+0000 7f6ded435700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f6de80fd770 0x7f6de81171c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:08.869 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.861+0000 7f6ded435700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de8117800 con 0x7f6de80fcde0 2023-06-22T17:26:08.869 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.861+0000 7f6ded435700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de8117970 con 0x7f6de80fd770 2023-06-22T17:26:08.869 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.861+0000 7f6ded435700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de8117b30 con 0x7f6de80fbbd0 2023-06-22T17:26:08.869 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.861+0000 7f6de67fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6de80fcde0 0x7f6de8110170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:08.870 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.861+0000 7f6de77fe700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f6de80fd770 0x7f6de81171c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:08.870 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.861+0000 7f6de77fe700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f6de80fd770 0x7f6de81171c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.39:56224/0 (socket says 172.21.15.39:56224) 2023-06-22T17:26:08.870 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.861+0000 7f6de67fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6de80fcde0 0x7f6de8110170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49352/0 (socket says 172.21.15.39:49352) 2023-06-22T17:26:08.870 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.861+0000 7f6de77fe700 1 -- 172.21.15.39:0/797192959 learned_addr learned my addr 172.21.15.39:0/797192959 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:26:08.870 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.861+0000 7f6de6ffd700 1 --2- 172.21.15.39:0/797192959 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6de80fbbd0 0x7f6de810fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:08.870 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.862+0000 7f6de77fe700 1 -- 172.21.15.39:0/797192959 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6de80fbbd0 msgr2=0x7f6de810fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:08.870 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.862+0000 7f6de77fe700 1 --2- 172.21.15.39:0/797192959 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6de80fbbd0 0x7f6de810fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:08.871 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.862+0000 7f6de77fe700 1 -- 172.21.15.39:0/797192959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6de80fcde0 msgr2=0x7f6de8110170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:08.871 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.862+0000 7f6de77fe700 1 --2- 172.21.15.39:0/797192959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6de80fcde0 0x7f6de8110170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:08.871 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.862+0000 7f6de77fe700 1 -- 172.21.15.39:0/797192959 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd0005040 con 0x7f6de80fd770 2023-06-22T17:26:08.871 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.862+0000 7f6de77fe700 1 --2- 172.21.15.39:0/797192959 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f6de80fd770 0x7f6de81171c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6de000ec30 tx=0x7f6de000b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:08.871 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.862+0000 7f6ddf7fe700 1 -- 172.21.15.39:0/797192959 <== mon.2 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de0007b30 con 0x7f6de80fd770 2023-06-22T17:26:08.871 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.862+0000 7f6ddf7fe700 1 -- 172.21.15.39:0/797192959 <== mon.2 v2:172.21.15.101:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f6de000f040 con 0x7f6de80fd770 2023-06-22T17:26:08.871 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.862+0000 7f6ded435700 1 -- 172.21.15.39:0/797192959 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6de8117e10 con 0x7f6de80fd770 2023-06-22T17:26:08.872 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.862+0000 7f6ddf7fe700 1 -- 172.21.15.39:0/797192959 <== mon.2 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de000caf0 con 0x7f6de80fd770 2023-06-22T17:26:08.872 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.862+0000 7f6ded435700 1 -- 172.21.15.39:0/797192959 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6de8118290 con 0x7f6de80fd770 2023-06-22T17:26:08.872 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.863+0000 7f6ded435700 1 -- 172.21.15.39:0/797192959 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6de81019e0 con 0x7f6de80fd770 2023-06-22T17:26:08.872 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.864+0000 7f6ddf7fe700 1 -- 172.21.15.39:0/797192959 <== mon.2 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f6de0043bb0 con 0x7f6de80fd770 2023-06-22T17:26:08.872 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.865+0000 7f6ddf7fe700 1 --2- 172.21.15.39:0/797192959 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f6dd409b680 0x7f6dd409db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:08.872 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.865+0000 7f6de6ffd700 1 --2- 172.21.15.39:0/797192959 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f6dd409b680 0x7f6dd409db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:08.872 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.865+0000 7f6ddf7fe700 1 -- 172.21.15.39:0/797192959 <== mon.2 v2:172.21.15.101:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f6de00c4d00 con 0x7f6de80fd770 2023-06-22T17:26:08.873 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.865+0000 7f6de6ffd700 1 --2- 172.21.15.39:0/797192959 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f6dd409b680 0x7f6dd409db30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6dd00099e0 tx=0x7f6dd0009e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:08.873 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:08.868+0000 7f6ddf7fe700 1 -- 172.21.15.39:0/797192959 <== mon.2 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f6de00909e0 con 0x7f6de80fd770 2023-06-22T17:26:09.112 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:09.110+0000 7f6ded435700 1 -- 172.21.15.39:0/797192959 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6de802b120 con 0x7f6de80fd770 2023-06-22T17:26:09.112 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:09.110+0000 7f6ddf7fe700 1 -- 172.21.15.39:0/797192959 <== mon.2 v2:172.21.15.101:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3531 (secure 0 0 0) 0x7f6de0043e60 con 0x7f6de80fd770 2023-06-22T17:26:09.113 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:26:09.113 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3630756954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3630756954},{"type":"v1","addr":"172.21.15.39:6835","nonce":3630756954}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/768662129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":768662129},{"type":"v1","addr":"172.21.15.39:6837","nonce":768662129}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6833/3120964256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":3120964256},{"type":"v1","addr":"172.21.15.101:6833","nonce":3120964256}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/1691506923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":1691506923},{"type":"v1","addr":"172.21.15.133:6833","nonce":1691506923}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-06-22T17:26:09.114 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:09.112+0000 7f6ded435700 1 -- 172.21.15.39:0/797192959 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f6dd409b680 msgr2=0x7f6dd409db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:09.115 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:09.112+0000 7f6ded435700 1 --2- 172.21.15.39:0/797192959 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f6dd409b680 0x7f6dd409db30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6dd00099e0 tx=0x7f6dd0009e30 comp rx=0 tx=0).stop 2023-06-22T17:26:09.115 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:09.112+0000 7f6ded435700 1 -- 172.21.15.39:0/797192959 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f6de80fd770 msgr2=0x7f6de81171c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:09.115 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:09.112+0000 7f6ded435700 1 --2- 172.21.15.39:0/797192959 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f6de80fd770 0x7f6de81171c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6de000ec30 tx=0x7f6de000b040 comp rx=0 tx=0).stop 2023-06-22T17:26:09.115 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:09.112+0000 7f6ded435700 1 -- 172.21.15.39:0/797192959 shutdown_connections 2023-06-22T17:26:09.115 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:09.112+0000 7f6ded435700 1 --2- 172.21.15.39:0/797192959 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f6dd409b680 0x7f6dd409db30 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:09.115 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:09.112+0000 7f6ded435700 1 --2- 172.21.15.39:0/797192959 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f6de80fd770 0x7f6de81171c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:09.116 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:09.112+0000 7f6ded435700 1 --2- 172.21.15.39:0/797192959 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6de80fbbd0 0x7f6de810fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:09.116 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:09.112+0000 7f6ded435700 1 --2- 172.21.15.39:0/797192959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6de80fcde0 0x7f6de8110170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:09.116 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:09.112+0000 7f6ded435700 1 -- 172.21.15.39:0/797192959 >> 172.21.15.39:0/797192959 conn(0x7f6de80f7310 msgr2=0x7f6de80f94d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:09.116 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:09.112+0000 7f6ded435700 1 -- 172.21.15.39:0/797192959 shutdown_connections 2023-06-22T17:26:09.116 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:09.112+0000 7f6ded435700 1 -- 172.21.15.39:0/797192959 wait complete. 2023-06-22T17:26:09.117 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 5 2023-06-22T17:26:09.126 INFO:journalctl@ceph.mds.d.smithi039.stdout:Jun 22 17:26:08 smithi039 ceph-mds[128638]: -- 172.21.15.39:0/647022962 <== mon.0 v2:172.21.15.39:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55745c1669c0 con 0x55745c13cc00 2023-06-22T17:26:09.126 INFO:journalctl@ceph.mds.d.smithi039.stdout:Jun 22 17:26:08 smithi039 conmon[128614]: starting mds.d at 2023-06-22T17:26:09.126 INFO:journalctl@ceph.mds.d.smithi039.stdout:Jun 22 17:26:08 smithi039 bash[128574]: e4e01f00e848bebae7368d0e112cc84d1cce0f110f81198bd6cf79acf7f9501a 2023-06-22T17:26:09.126 INFO:journalctl@ceph.mds.d.smithi039.stdout:Jun 22 17:26:08 smithi039 systemd[1]: Started Ceph mds.d for d88002e2-1120-11ee-9b2b-001a4aab830c. 2023-06-22T17:26:10.122 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:09 smithi101 ceph-mon[85507]: mds.? [v2:172.21.15.39:6836/768662129,v1:172.21.15.39:6837/768662129] up:boot 2023-06-22T17:26:10.122 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:09 smithi101 ceph-mon[85507]: fsmap 4 up:standby 2023-06-22T17:26:10.122 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:09 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-06-22T17:26:10.123 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:09 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:10.123 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:09 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:10.123 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:09 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:10.123 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:09 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:10.123 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:09 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:10.123 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:09 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:10.123 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:09 smithi101 ceph-mon[85507]: Deploying daemon mds.e on smithi101 2023-06-22T17:26:10.124 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:09 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/797192959' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:10.124 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:09 smithi101 ceph-mon[85507]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:10.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:09 smithi133 ceph-mon[86268]: mds.? [v2:172.21.15.39:6836/768662129,v1:172.21.15.39:6837/768662129] up:boot 2023-06-22T17:26:10.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:09 smithi133 ceph-mon[86268]: fsmap 4 up:standby 2023-06-22T17:26:10.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:09 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-06-22T17:26:10.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:09 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:10.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:09 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:10.172 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:09 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:10.172 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:09 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:10.172 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:09 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:10.172 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:09 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:10.172 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:09 smithi133 ceph-mon[86268]: Deploying daemon mds.e on smithi101 2023-06-22T17:26:10.172 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:09 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/797192959' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:10.172 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:09 smithi133 ceph-mon[86268]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:10.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:09 smithi039 ceph-mon[84805]: mds.? [v2:172.21.15.39:6836/768662129,v1:172.21.15.39:6837/768662129] up:boot 2023-06-22T17:26:10.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:09 smithi039 ceph-mon[84805]: fsmap 4 up:standby 2023-06-22T17:26:10.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:09 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-06-22T17:26:10.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:09 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:10.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:09 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:10.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:09 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:10.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:09 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:10.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:09 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:10.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:09 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:10.185 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:09 smithi039 ceph-mon[84805]: Deploying daemon mds.e on smithi101 2023-06-22T17:26:10.185 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:09 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/797192959' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:10.185 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:09 smithi039 ceph-mon[84805]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:10.376 INFO:journalctl@ceph.mds.e.smithi101.stdout:Jun 22 17:26:10 smithi101 systemd[1]: Starting Ceph mds.e for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:26:11.126 INFO:journalctl@ceph.mds.e.smithi101.stdout:Jun 22 17:26:10 smithi101 ceph-mds[117067]: -- 172.21.15.101:0/2557231268 <== mon.2 v2:172.21.15.101:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a08f23c9c0 con 0x55a08f212800 2023-06-22T17:26:11.127 INFO:journalctl@ceph.mds.e.smithi101.stdout:Jun 22 17:26:10 smithi101 conmon[117043]: starting mds.e at 2023-06-22T17:26:11.127 INFO:journalctl@ceph.mds.e.smithi101.stdout:Jun 22 17:26:10 smithi101 bash[117003]: 5b011b0a25280afc2dc997d79f7a6006fd0a558033cda5540564f57d838b726b 2023-06-22T17:26:11.127 INFO:journalctl@ceph.mds.e.smithi101.stdout:Jun 22 17:26:10 smithi101 systemd[1]: Started Ceph mds.e for d88002e2-1120-11ee-9b2b-001a4aab830c. 2023-06-22T17:26:11.128 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T17:26:11.261 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.259+0000 7fc8e003f700 1 -- 172.21.15.39:0/2400258025 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8d80fd1f0 msgr2=0x7fc8d80fd600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:11.261 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.259+0000 7fc8e003f700 1 --2- 172.21.15.39:0/2400258025 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8d80fd1f0 0x7fc8d80fd600 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc8c80049c0 tx=0x7fc8c8009e30 comp rx=0 tx=0).stop 2023-06-22T17:26:11.262 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.259+0000 7fc8e003f700 1 -- 172.21.15.39:0/2400258025 shutdown_connections 2023-06-22T17:26:11.262 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.259+0000 7fc8e003f700 1 --2- 172.21.15.39:0/2400258025 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8d80fc860 0x7fc8d8104110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:11.262 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.259+0000 7fc8e003f700 1 --2- 172.21.15.39:0/2400258025 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8d80fd1f0 0x7fc8d80fd600 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:11.262 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.259+0000 7fc8e003f700 1 --2- 172.21.15.39:0/2400258025 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc8d80fbeb0 0x7fc8d80fc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:11.262 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.259+0000 7fc8e003f700 1 -- 172.21.15.39:0/2400258025 >> 172.21.15.39:0/2400258025 conn(0x7fc8d80f6ab0 msgr2=0x7fc8d80f8f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:11.262 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.259+0000 7fc8e003f700 1 -- 172.21.15.39:0/2400258025 shutdown_connections 2023-06-22T17:26:11.262 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.259+0000 7fc8e003f700 1 -- 172.21.15.39:0/2400258025 wait complete. 2023-06-22T17:26:11.264 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.260+0000 7fc8e003f700 1 Processor -- start 2023-06-22T17:26:11.264 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.260+0000 7fc8e003f700 1 -- start start 2023-06-22T17:26:11.264 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.260+0000 7fc8e003f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8d80fbeb0 0x7fc8d810c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:11.265 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.260+0000 7fc8e003f700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8d80fc860 0x7fc8d810ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:11.265 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.260+0000 7fc8e003f700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc8d80fd1f0 0x7fc8d8113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:11.265 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.260+0000 7fc8e003f700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d81144e0 con 0x7fc8d80fc860 2023-06-22T17:26:11.265 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.260+0000 7fc8e003f700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d8114650 con 0x7fc8d80fd1f0 2023-06-22T17:26:11.265 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.260+0000 7fc8e003f700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d8114810 con 0x7fc8d80fbeb0 2023-06-22T17:26:11.265 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.260+0000 7fc8dd5da700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8d80fc860 0x7fc8d810ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:11.266 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.260+0000 7fc8dd5da700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8d80fc860 0x7fc8d810ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49368/0 (socket says 172.21.15.39:49368) 2023-06-22T17:26:11.266 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.260+0000 7fc8dd5da700 1 -- 172.21.15.39:0/42263718 learned_addr learned my addr 172.21.15.39:0/42263718 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:26:11.266 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.260+0000 7fc8ddddb700 1 --2- 172.21.15.39:0/42263718 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8d80fbeb0 0x7fc8d810c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:11.266 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.260+0000 7fc8de5dc700 1 --2- 172.21.15.39:0/42263718 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc8d80fd1f0 0x7fc8d8113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:11.266 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.260+0000 7fc8dd5da700 1 -- 172.21.15.39:0/42263718 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8d80fbeb0 msgr2=0x7fc8d810c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:11.266 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.260+0000 7fc8dd5da700 1 --2- 172.21.15.39:0/42263718 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8d80fbeb0 0x7fc8d810c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:11.266 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.260+0000 7fc8dd5da700 1 -- 172.21.15.39:0/42263718 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc8d80fd1f0 msgr2=0x7fc8d8113ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:11.266 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.260+0000 7fc8dd5da700 1 --2- 172.21.15.39:0/42263718 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc8d80fd1f0 0x7fc8d8113ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:11.267 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.261+0000 7fc8dd5da700 1 -- 172.21.15.39:0/42263718 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c8005040 con 0x7fc8d80fc860 2023-06-22T17:26:11.267 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.261+0000 7fc8dd5da700 1 --2- 172.21.15.39:0/42263718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8d80fc860 0x7fc8d810ce40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc8d4002b00 tx=0x7fc8d40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:11.267 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.261+0000 7fc8ce7fc700 1 -- 172.21.15.39:0/42263718 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d4007cc0 con 0x7fc8d80fc860 2023-06-22T17:26:11.267 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.261+0000 7fc8e003f700 1 -- 172.21.15.39:0/42263718 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8d8114af0 con 0x7fc8d80fc860 2023-06-22T17:26:11.267 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.261+0000 7fc8ce7fc700 1 -- 172.21.15.39:0/42263718 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7fc8d400f040 con 0x7fc8d80fc860 2023-06-22T17:26:11.267 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.261+0000 7fc8ce7fc700 1 -- 172.21.15.39:0/42263718 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d400d9f0 con 0x7fc8d80fc860 2023-06-22T17:26:11.267 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.261+0000 7fc8e003f700 1 -- 172.21.15.39:0/42263718 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8d8115030 con 0x7fc8d80fc860 2023-06-22T17:26:11.267 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.262+0000 7fc8e003f700 1 -- 172.21.15.39:0/42263718 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8d8107550 con 0x7fc8d80fc860 2023-06-22T17:26:11.268 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.263+0000 7fc8ce7fc700 1 -- 172.21.15.39:0/42263718 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7fc8d4009090 con 0x7fc8d80fc860 2023-06-22T17:26:11.268 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.264+0000 7fc8ce7fc700 1 --2- 172.21.15.39:0/42263718 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fc8c409b680 0x7fc8c409db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:11.269 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.264+0000 7fc8ddddb700 1 --2- 172.21.15.39:0/42263718 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fc8c409b680 0x7fc8c409db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:11.269 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.264+0000 7fc8ce7fc700 1 -- 172.21.15.39:0/42263718 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fc8d40c4b00 con 0x7fc8d80fc860 2023-06-22T17:26:11.269 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.264+0000 7fc8ddddb700 1 --2- 172.21.15.39:0/42263718 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fc8c409b680 0x7fc8c409db30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8009f00 tx=0x7fc8c8009e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:11.269 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.267+0000 7fc8ce7fc700 1 -- 172.21.15.39:0/42263718 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fc8d4090760 con 0x7fc8d80fc860 2023-06-22T17:26:11.511 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.509+0000 7fc8e003f700 1 -- 172.21.15.39:0/42263718 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8d81076e0 con 0x7fc8d80fc860 2023-06-22T17:26:11.511 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.509+0000 7fc8ce7fc700 1 -- 172.21.15.39:0/42263718 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4291 (secure 0 0 0) 0x7fc8d81076e0 con 0x7fc8d80fc860 2023-06-22T17:26:11.512 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:26:11.512 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3630756954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3630756954},{"type":"v1","addr":"172.21.15.39:6835","nonce":3630756954}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/768662129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":768662129},{"type":"v1","addr":"172.21.15.39:6837","nonce":768662129}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6833/3120964256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":3120964256},{"type":"v1","addr":"172.21.15.101:6833","nonce":3120964256}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/1691506923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":1691506923},{"type":"v1","addr":"172.21.15.133:6833","nonce":1691506923}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6835/4021581510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":4021581510},{"type":"v1","addr":"172.21.15.101:6835","nonce":4021581510}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2023-06-22T17:26:11.514 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.512+0000 7fc8e003f700 1 -- 172.21.15.39:0/42263718 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fc8c409b680 msgr2=0x7fc8c409db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:11.514 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.512+0000 7fc8e003f700 1 --2- 172.21.15.39:0/42263718 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fc8c409b680 0x7fc8c409db30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8009f00 tx=0x7fc8c8009e30 comp rx=0 tx=0).stop 2023-06-22T17:26:11.514 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.512+0000 7fc8e003f700 1 -- 172.21.15.39:0/42263718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8d80fc860 msgr2=0x7fc8d810ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:11.514 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.512+0000 7fc8e003f700 1 --2- 172.21.15.39:0/42263718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8d80fc860 0x7fc8d810ce40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc8d4002b00 tx=0x7fc8d40046b0 comp rx=0 tx=0).stop 2023-06-22T17:26:11.514 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.512+0000 7fc8e003f700 1 -- 172.21.15.39:0/42263718 shutdown_connections 2023-06-22T17:26:11.514 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.512+0000 7fc8e003f700 1 --2- 172.21.15.39:0/42263718 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fc8c409b680 0x7fc8c409db30 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:11.515 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.512+0000 7fc8e003f700 1 --2- 172.21.15.39:0/42263718 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc8d80fd1f0 0x7fc8d8113ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:11.515 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.512+0000 7fc8e003f700 1 --2- 172.21.15.39:0/42263718 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8d80fbeb0 0x7fc8d810c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:11.515 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.512+0000 7fc8e003f700 1 --2- 172.21.15.39:0/42263718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8d80fc860 0x7fc8d810ce40 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:11.515 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.512+0000 7fc8e003f700 1 -- 172.21.15.39:0/42263718 >> 172.21.15.39:0/42263718 conn(0x7fc8d80f6ab0 msgr2=0x7fc8d80f8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:11.515 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.512+0000 7fc8e003f700 1 -- 172.21.15.39:0/42263718 shutdown_connections 2023-06-22T17:26:11.515 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:11.512+0000 7fc8e003f700 1 -- 172.21.15.39:0/42263718 wait complete. 2023-06-22T17:26:11.516 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 6 2023-06-22T17:26:12.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:11 smithi133 ceph-mon[86268]: mds.? [v2:172.21.15.101:6834/4021581510,v1:172.21.15.101:6835/4021581510] up:boot 2023-06-22T17:26:12.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:11 smithi133 ceph-mon[86268]: fsmap 5 up:standby 2023-06-22T17:26:12.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:11 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-06-22T17:26:12.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:11 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:12.172 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:11 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:12.172 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:11 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:12.172 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:11 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:12.172 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:11 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:12.172 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:11 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:12.172 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:11 smithi133 ceph-mon[86268]: Deploying daemon mds.f on smithi133 2023-06-22T17:26:12.172 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:11 smithi133 ceph-mon[86268]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:12.172 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:11 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/42263718' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:12.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:11 smithi039 ceph-mon[84805]: mds.? [v2:172.21.15.101:6834/4021581510,v1:172.21.15.101:6835/4021581510] up:boot 2023-06-22T17:26:12.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:11 smithi039 ceph-mon[84805]: fsmap 5 up:standby 2023-06-22T17:26:12.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:11 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-06-22T17:26:12.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:11 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:12.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:11 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:12.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:11 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:12.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:11 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:12.185 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:11 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:12.185 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:11 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:12.185 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:11 smithi039 ceph-mon[84805]: Deploying daemon mds.f on smithi133 2023-06-22T17:26:12.185 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:11 smithi039 ceph-mon[84805]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:12.185 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:11 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/42263718' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:12.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:11 smithi101 ceph-mon[85507]: mds.? [v2:172.21.15.101:6834/4021581510,v1:172.21.15.101:6835/4021581510] up:boot 2023-06-22T17:26:12.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:11 smithi101 ceph-mon[85507]: fsmap 5 up:standby 2023-06-22T17:26:12.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-06-22T17:26:12.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:12.378 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:12.378 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:12.378 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:12.378 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:12.379 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:11 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:12.379 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:11 smithi101 ceph-mon[85507]: Deploying daemon mds.f on smithi133 2023-06-22T17:26:12.379 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:11 smithi101 ceph-mon[85507]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:12.379 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:11 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/42263718' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:12.921 INFO:journalctl@ceph.mds.f.smithi133.stdout:Jun 22 17:26:12 smithi133 systemd[1]: Starting Ceph mds.f for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:26:13.527 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T17:26:13.663 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.661+0000 7faca59b8700 1 -- 172.21.15.39:0/929430893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faca00ffc90 msgr2=0x7faca0102070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:13.663 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.661+0000 7faca59b8700 1 --2- 172.21.15.39:0/929430893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faca00ffc90 0x7faca0102070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fac880049c0 tx=0x7fac88009e30 comp rx=0 tx=0).stop 2023-06-22T17:26:13.664 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.661+0000 7faca59b8700 1 -- 172.21.15.39:0/929430893 shutdown_connections 2023-06-22T17:26:13.664 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.661+0000 7faca59b8700 1 --2- 172.21.15.39:0/929430893 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faca0102780 0x7faca0104c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:13.664 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.661+0000 7faca59b8700 1 --2- 172.21.15.39:0/929430893 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faca0065c10 0x7faca00ff750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:13.664 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.661+0000 7faca59b8700 1 --2- 172.21.15.39:0/929430893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faca00ffc90 0x7faca0102070 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:13.664 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.661+0000 7faca59b8700 1 -- 172.21.15.39:0/929430893 >> 172.21.15.39:0/929430893 conn(0x7faca00f7120 msgr2=0x7faca00f9590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:13.664 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.661+0000 7faca59b8700 1 -- 172.21.15.39:0/929430893 shutdown_connections 2023-06-22T17:26:13.664 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.661+0000 7faca59b8700 1 -- 172.21.15.39:0/929430893 wait complete. 2023-06-22T17:26:13.666 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.662+0000 7faca59b8700 1 Processor -- start 2023-06-22T17:26:13.666 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.662+0000 7faca59b8700 1 -- start start 2023-06-22T17:26:13.667 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.662+0000 7faca59b8700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faca0065c10 0x7faca010d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:13.667 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.662+0000 7faca59b8700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faca00ffc90 0x7faca010dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:13.667 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.662+0000 7faca59b8700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faca0102780 0x7faca0114da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:13.667 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.662+0000 7faca59b8700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca01153b0 con 0x7faca0102780 2023-06-22T17:26:13.667 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.662+0000 7faca59b8700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca0115520 con 0x7faca0065c10 2023-06-22T17:26:13.667 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.662+0000 7faca59b8700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca01156e0 con 0x7faca00ffc90 2023-06-22T17:26:13.667 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.662+0000 7fac9f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faca0102780 0x7faca0114da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:13.668 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.662+0000 7fac9f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faca0102780 0x7faca0114da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49392/0 (socket says 172.21.15.39:49392) 2023-06-22T17:26:13.668 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.662+0000 7fac9f7fe700 1 -- 172.21.15.39:0/1160528649 learned_addr learned my addr 172.21.15.39:0/1160528649 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:26:13.668 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.662+0000 7fac9e7fc700 1 --2- 172.21.15.39:0/1160528649 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faca00ffc90 0x7faca010dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:13.668 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.662+0000 7fac9effd700 1 --2- 172.21.15.39:0/1160528649 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faca0065c10 0x7faca010d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:13.668 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.662+0000 7fac9f7fe700 1 -- 172.21.15.39:0/1160528649 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faca00ffc90 msgr2=0x7faca010dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:13.668 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.662+0000 7fac9f7fe700 1 --2- 172.21.15.39:0/1160528649 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faca00ffc90 0x7faca010dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:13.668 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.662+0000 7fac9f7fe700 1 -- 172.21.15.39:0/1160528649 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faca0065c10 msgr2=0x7faca010d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:13.668 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.662+0000 7fac9f7fe700 1 --2- 172.21.15.39:0/1160528649 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faca0065c10 0x7faca010d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:13.669 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.662+0000 7fac9f7fe700 1 -- 172.21.15.39:0/1160528649 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac88005040 con 0x7faca0102780 2023-06-22T17:26:13.669 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.663+0000 7fac9f7fe700 1 --2- 172.21.15.39:0/1160528649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faca0102780 0x7faca0114da0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fac9000d1d0 tx=0x7fac90004d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:13.669 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.663+0000 7fac977fe700 1 -- 172.21.15.39:0/1160528649 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9000dd10 con 0x7faca0102780 2023-06-22T17:26:13.669 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.663+0000 7fac977fe700 1 -- 172.21.15.39:0/1160528649 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7fac9000f040 con 0x7faca0102780 2023-06-22T17:26:13.669 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.663+0000 7faca59b8700 1 -- 172.21.15.39:0/1160528649 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faca01159c0 con 0x7faca0102780 2023-06-22T17:26:13.669 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.663+0000 7fac977fe700 1 -- 172.21.15.39:0/1160528649 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac90009320 con 0x7faca0102780 2023-06-22T17:26:13.669 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.663+0000 7faca59b8700 1 -- 172.21.15.39:0/1160528649 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faca0115ee0 con 0x7faca0102780 2023-06-22T17:26:13.670 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.664+0000 7faca59b8700 1 -- 172.21.15.39:0/1160528649 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faca00f8c50 con 0x7faca0102780 2023-06-22T17:26:13.670 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.665+0000 7fac977fe700 1 -- 172.21.15.39:0/1160528649 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7fac90009750 con 0x7faca0102780 2023-06-22T17:26:13.670 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.665+0000 7fac977fe700 1 --2- 172.21.15.39:0/1160528649 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fac8c09b680 0x7fac8c09db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:13.671 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.666+0000 7fac977fe700 1 -- 172.21.15.39:0/1160528649 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fac900c51a0 con 0x7faca0102780 2023-06-22T17:26:13.671 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.666+0000 7fac9effd700 1 --2- 172.21.15.39:0/1160528649 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fac8c09b680 0x7fac8c09db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:13.671 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.666+0000 7fac9effd700 1 --2- 172.21.15.39:0/1160528649 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fac8c09b680 0x7fac8c09db30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7faca0065950 tx=0x7fac9800b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:13.671 INFO:journalctl@ceph.mds.f.smithi133.stdout:Jun 22 17:26:13 smithi133 ceph-mds[119473]: -- 172.21.15.133:0/8180907 <== mon.1 v2:172.21.15.133:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5586905b69c0 con 0x55869058cc00 2023-06-22T17:26:13.671 INFO:journalctl@ceph.mds.f.smithi133.stdout:Jun 22 17:26:13 smithi133 conmon[119449]: starting mds.f at 2023-06-22T17:26:13.671 INFO:journalctl@ceph.mds.f.smithi133.stdout:Jun 22 17:26:13 smithi133 bash[119409]: 58e0ea00e111f119d07de51c2853c21b3d52afa23a9dc9b6acf07c3882604a39 2023-06-22T17:26:13.672 INFO:journalctl@ceph.mds.f.smithi133.stdout:Jun 22 17:26:13 smithi133 systemd[1]: Started Ceph mds.f for d88002e2-1120-11ee-9b2b-001a4aab830c. 2023-06-22T17:26:13.672 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.670+0000 7fac977fe700 1 -- 172.21.15.39:0/1160528649 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fac90090f30 con 0x7faca0102780 2023-06-22T17:26:13.913 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.911+0000 7faca59b8700 1 -- 172.21.15.39:0/1160528649 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faca002b1f0 con 0x7faca0102780 2023-06-22T17:26:13.914 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.912+0000 7fac977fe700 1 -- 172.21.15.39:0/1160528649 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5051 (secure 0 0 0) 0x7fac90094550 con 0x7faca0102780 2023-06-22T17:26:13.914 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:26:13.914 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3630756954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3630756954},{"type":"v1","addr":"172.21.15.39:6835","nonce":3630756954}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/768662129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":768662129},{"type":"v1","addr":"172.21.15.39:6837","nonce":768662129}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6833/3120964256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":3120964256},{"type":"v1","addr":"172.21.15.101:6833","nonce":3120964256}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/1691506923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":1691506923},{"type":"v1","addr":"172.21.15.133:6833","nonce":1691506923}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6835/4021581510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":4021581510},{"type":"v1","addr":"172.21.15.101:6835","nonce":4021581510}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/1920373312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":1920373312},{"type":"v1","addr":"172.21.15.133:6835","nonce":1920373312}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[]} 2023-06-22T17:26:13.916 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.914+0000 7faca59b8700 1 -- 172.21.15.39:0/1160528649 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fac8c09b680 msgr2=0x7fac8c09db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:13.916 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.914+0000 7faca59b8700 1 --2- 172.21.15.39:0/1160528649 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fac8c09b680 0x7fac8c09db30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7faca0065950 tx=0x7fac9800b040 comp rx=0 tx=0).stop 2023-06-22T17:26:13.916 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.914+0000 7faca59b8700 1 -- 172.21.15.39:0/1160528649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faca0102780 msgr2=0x7faca0114da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:13.916 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.914+0000 7faca59b8700 1 --2- 172.21.15.39:0/1160528649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faca0102780 0x7faca0114da0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fac9000d1d0 tx=0x7fac90004d70 comp rx=0 tx=0).stop 2023-06-22T17:26:13.917 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.914+0000 7faca59b8700 1 -- 172.21.15.39:0/1160528649 shutdown_connections 2023-06-22T17:26:13.917 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.914+0000 7faca59b8700 1 --2- 172.21.15.39:0/1160528649 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fac8c09b680 0x7fac8c09db30 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:13.917 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.914+0000 7faca59b8700 1 --2- 172.21.15.39:0/1160528649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faca0102780 0x7faca0114da0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:13.917 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.914+0000 7faca59b8700 1 --2- 172.21.15.39:0/1160528649 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faca0065c10 0x7faca010d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:13.917 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.914+0000 7faca59b8700 1 --2- 172.21.15.39:0/1160528649 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faca00ffc90 0x7faca010dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:13.917 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.914+0000 7faca59b8700 1 -- 172.21.15.39:0/1160528649 >> 172.21.15.39:0/1160528649 conn(0x7faca00f7120 msgr2=0x7faca00fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:13.917 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.914+0000 7faca59b8700 1 -- 172.21.15.39:0/1160528649 shutdown_connections 2023-06-22T17:26:13.918 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:13.914+0000 7faca59b8700 1 -- 172.21.15.39:0/1160528649 wait complete. 2023-06-22T17:26:13.918 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 7 2023-06-22T17:26:14.545 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:14 smithi133 ceph-mon[86268]: mds.? [v2:172.21.15.133:6834/1920373312,v1:172.21.15.133:6835/1920373312] up:boot 2023-06-22T17:26:14.545 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:14 smithi133 ceph-mon[86268]: fsmap 6 up:standby 2023-06-22T17:26:14.546 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:14 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-06-22T17:26:14.546 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:14 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:14.546 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:14 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:14.546 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:14 smithi133 ceph-mon[86268]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:14.546 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:14 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:14.546 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:14 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:14.546 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:14 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:14.547 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:14 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:14.547 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:14 smithi133 ceph-mon[86268]: Deploying daemon mds.c on smithi133 2023-06-22T17:26:14.547 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:14 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/1160528649' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:14.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:14 smithi101 ceph-mon[85507]: mds.? [v2:172.21.15.133:6834/1920373312,v1:172.21.15.133:6835/1920373312] up:boot 2023-06-22T17:26:14.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:14 smithi101 ceph-mon[85507]: fsmap 6 up:standby 2023-06-22T17:26:14.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:14 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-06-22T17:26:14.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:14 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:14.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:14 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:14.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:14 smithi101 ceph-mon[85507]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:14.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:14 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:14.628 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:14 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:14.628 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:14 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:14.628 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:14 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:14.628 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:14 smithi101 ceph-mon[85507]: Deploying daemon mds.c on smithi133 2023-06-22T17:26:14.628 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:14 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/1160528649' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:14.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:14 smithi039 ceph-mon[84805]: mds.? [v2:172.21.15.133:6834/1920373312,v1:172.21.15.133:6835/1920373312] up:boot 2023-06-22T17:26:14.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:14 smithi039 ceph-mon[84805]: fsmap 6 up:standby 2023-06-22T17:26:14.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:14 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-06-22T17:26:14.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:14 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:14.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:14 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:14.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:14 smithi039 ceph-mon[84805]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:14.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:14 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:14.685 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:14 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:14.685 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:14 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:14.685 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:14 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:14.685 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:14 smithi039 ceph-mon[84805]: Deploying daemon mds.c on smithi133 2023-06-22T17:26:14.685 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:14 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/1160528649' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:14.833 INFO:journalctl@ceph.mds.c.smithi133.stdout:Jun 22 17:26:14 smithi133 systemd[1]: Starting Ceph mds.c for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:26:15.920 INFO:journalctl@ceph.mds.c.smithi133.stdout:Jun 22 17:26:15 smithi133 ceph-mds[120035]: -- 172.21.15.133:0/3177524966 <== mon.1 v2:172.21.15.133:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556a8045e9c0 con 0x556a80435000 2023-06-22T17:26:15.921 INFO:journalctl@ceph.mds.c.smithi133.stdout:Jun 22 17:26:15 smithi133 conmon[120011]: starting mds.c at 2023-06-22T17:26:15.921 INFO:journalctl@ceph.mds.c.smithi133.stdout:Jun 22 17:26:15 smithi133 bash[119970]: 214675f4859dd83877e256adb92bd4c7f66602caa1eacd11a4d650ed4a58e92b 2023-06-22T17:26:15.921 INFO:journalctl@ceph.mds.c.smithi133.stdout:Jun 22 17:26:15 smithi133 systemd[1]: Started Ceph mds.c for d88002e2-1120-11ee-9b2b-001a4aab830c. 2023-06-22T17:26:15.930 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T17:26:16.064 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.062+0000 7f04b1ba2700 1 -- 172.21.15.39:0/4203888304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04ac0fb9e0 msgr2=0x7f04ac0fbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:16.065 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.062+0000 7f04b1ba2700 1 --2- 172.21.15.39:0/4203888304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04ac0fb9e0 0x7f04ac0fbdf0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f049c0049c0 tx=0x7f049c009e30 comp rx=0 tx=0).stop 2023-06-22T17:26:16.065 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.062+0000 7f04b1ba2700 1 -- 172.21.15.39:0/4203888304 shutdown_connections 2023-06-22T17:26:16.065 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.062+0000 7f04b1ba2700 1 --2- 172.21.15.39:0/4203888304 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f04ac0fd640 0x7f04ac103ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:16.066 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.062+0000 7f04b1ba2700 1 --2- 172.21.15.39:0/4203888304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04ac0fb9e0 0x7f04ac0fbdf0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:16.066 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.062+0000 7f04b1ba2700 1 --2- 172.21.15.39:0/4203888304 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f04ac0fcc90 0x7f04ac0fd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:16.066 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.062+0000 7f04b1ba2700 1 -- 172.21.15.39:0/4203888304 >> 172.21.15.39:0/4203888304 conn(0x7f04ac0f7140 msgr2=0x7f04ac0f9570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:16.066 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.062+0000 7f04b1ba2700 1 -- 172.21.15.39:0/4203888304 shutdown_connections 2023-06-22T17:26:16.066 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.062+0000 7f04b1ba2700 1 -- 172.21.15.39:0/4203888304 wait complete. 2023-06-22T17:26:16.066 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.063+0000 7f04b1ba2700 1 Processor -- start 2023-06-22T17:26:16.066 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.063+0000 7f04b1ba2700 1 -- start start 2023-06-22T17:26:16.068 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.063+0000 7f04b1ba2700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f04ac0fb9e0 0x7f04ac10c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:16.068 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.063+0000 7f04b1ba2700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04ac0fcc90 0x7f04ac10cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:16.069 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.063+0000 7f04b1ba2700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f04ac0fd640 0x7f04ac113c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:16.069 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.063+0000 7f04b1ba2700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac114270 con 0x7f04ac0fcc90 2023-06-22T17:26:16.069 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.063+0000 7f04b1ba2700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac1143e0 con 0x7f04ac0fd640 2023-06-22T17:26:16.069 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.063+0000 7f04b1ba2700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac1145a0 con 0x7f04ac0fb9e0 2023-06-22T17:26:16.069 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.063+0000 7f04abfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04ac0fcc90 0x7f04ac10cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:16.069 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.063+0000 7f04abfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04ac0fcc90 0x7f04ac10cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58980/0 (socket says 172.21.15.39:58980) 2023-06-22T17:26:16.069 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.063+0000 7f04abfff700 1 -- 172.21.15.39:0/223753308 learned_addr learned my addr 172.21.15.39:0/223753308 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:26:16.070 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.063+0000 7f04b0ba0700 1 --2- 172.21.15.39:0/223753308 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f04ac0fb9e0 0x7f04ac10c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:16.070 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.063+0000 7f04b13a1700 1 --2- 172.21.15.39:0/223753308 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f04ac0fd640 0x7f04ac113c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:16.070 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.064+0000 7f04abfff700 1 -- 172.21.15.39:0/223753308 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f04ac0fb9e0 msgr2=0x7f04ac10c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:16.070 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.064+0000 7f04abfff700 1 --2- 172.21.15.39:0/223753308 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f04ac0fb9e0 0x7f04ac10c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:16.070 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.064+0000 7f04abfff700 1 -- 172.21.15.39:0/223753308 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f04ac0fd640 msgr2=0x7f04ac113c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:16.070 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.064+0000 7f04abfff700 1 --2- 172.21.15.39:0/223753308 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f04ac0fd640 0x7f04ac113c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:16.070 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.064+0000 7f04abfff700 1 -- 172.21.15.39:0/223753308 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f049c005040 con 0x7f04ac0fcc90 2023-06-22T17:26:16.071 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.064+0000 7f04abfff700 1 --2- 172.21.15.39:0/223753308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04ac0fcc90 0x7f04ac10cbd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f04a000ebc0 tx=0x7f04a000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:16.071 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.064+0000 7f04a97fa700 1 -- 172.21.15.39:0/223753308 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a0007cc0 con 0x7f04ac0fcc90 2023-06-22T17:26:16.071 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.064+0000 7f04b1ba2700 1 -- 172.21.15.39:0/223753308 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04ac114820 con 0x7f04ac0fcc90 2023-06-22T17:26:16.071 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.064+0000 7f04a97fa700 1 -- 172.21.15.39:0/223753308 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f04a000f040 con 0x7f04ac0fcc90 2023-06-22T17:26:16.071 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.064+0000 7f04a97fa700 1 -- 172.21.15.39:0/223753308 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a000c930 con 0x7f04ac0fcc90 2023-06-22T17:26:16.071 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.064+0000 7f04b1ba2700 1 -- 172.21.15.39:0/223753308 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04ac114de0 con 0x7f04ac0fcc90 2023-06-22T17:26:16.071 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.065+0000 7f04b1ba2700 1 -- 172.21.15.39:0/223753308 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04ac102e60 con 0x7f04ac0fcc90 2023-06-22T17:26:16.071 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.066+0000 7f04a97fa700 1 -- 172.21.15.39:0/223753308 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f04a00090d0 con 0x7f04ac0fcc90 2023-06-22T17:26:16.072 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.067+0000 7f04a97fa700 1 --2- 172.21.15.39:0/223753308 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f049409b680 0x7f049409db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:16.072 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.067+0000 7f04b0ba0700 1 --2- 172.21.15.39:0/223753308 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f049409b680 0x7f049409db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:16.072 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.067+0000 7f04a97fa700 1 -- 172.21.15.39:0/223753308 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f04a00c4a80 con 0x7f04ac0fcc90 2023-06-22T17:26:16.073 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.067+0000 7f04b0ba0700 1 --2- 172.21.15.39:0/223753308 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f049409b680 0x7f049409db30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f049c009f00 tx=0x7f049c009e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:16.073 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.070+0000 7f04a97fa700 1 -- 172.21.15.39:0/223753308 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f04a0090780 con 0x7f04ac0fcc90 2023-06-22T17:26:16.315 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.313+0000 7f04b1ba2700 1 -- 172.21.15.39:0/223753308 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04ac107330 con 0x7f04ac0fcc90 2023-06-22T17:26:16.316 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.314+0000 7f04a97fa700 1 -- 172.21.15.39:0/223753308 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5051 (secure 0 0 0) 0x7f04a0093da0 con 0x7f04ac0fcc90 2023-06-22T17:26:16.316 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:26:16.316 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3630756954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3630756954},{"type":"v1","addr":"172.21.15.39:6835","nonce":3630756954}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/768662129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":768662129},{"type":"v1","addr":"172.21.15.39:6837","nonce":768662129}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6833/3120964256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":3120964256},{"type":"v1","addr":"172.21.15.101:6833","nonce":3120964256}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/1691506923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":1691506923},{"type":"v1","addr":"172.21.15.133:6833","nonce":1691506923}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6835/4021581510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":4021581510},{"type":"v1","addr":"172.21.15.101:6835","nonce":4021581510}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/1920373312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":1920373312},{"type":"v1","addr":"172.21.15.133:6835","nonce":1920373312}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[]} 2023-06-22T17:26:16.318 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.316+0000 7f04b1ba2700 1 -- 172.21.15.39:0/223753308 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f049409b680 msgr2=0x7f049409db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:16.318 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.316+0000 7f04b1ba2700 1 --2- 172.21.15.39:0/223753308 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f049409b680 0x7f049409db30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f049c009f00 tx=0x7f049c009e30 comp rx=0 tx=0).stop 2023-06-22T17:26:16.318 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.316+0000 7f04b1ba2700 1 -- 172.21.15.39:0/223753308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04ac0fcc90 msgr2=0x7f04ac10cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:16.318 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.316+0000 7f04b1ba2700 1 --2- 172.21.15.39:0/223753308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04ac0fcc90 0x7f04ac10cbd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f04a000ebc0 tx=0x7f04a000b040 comp rx=0 tx=0).stop 2023-06-22T17:26:16.318 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.316+0000 7f04b1ba2700 1 -- 172.21.15.39:0/223753308 shutdown_connections 2023-06-22T17:26:16.318 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.316+0000 7f04b1ba2700 1 --2- 172.21.15.39:0/223753308 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f049409b680 0x7f049409db30 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:16.319 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.316+0000 7f04b1ba2700 1 --2- 172.21.15.39:0/223753308 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f04ac0fd640 0x7f04ac113c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:16.319 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.316+0000 7f04b1ba2700 1 --2- 172.21.15.39:0/223753308 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f04ac0fb9e0 0x7f04ac10c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:16.319 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.316+0000 7f04b1ba2700 1 --2- 172.21.15.39:0/223753308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04ac0fcc90 0x7f04ac10cbd0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:16.319 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.316+0000 7f04b1ba2700 1 -- 172.21.15.39:0/223753308 >> 172.21.15.39:0/223753308 conn(0x7f04ac0f7140 msgr2=0x7f04ac0fffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:16.319 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.316+0000 7f04b1ba2700 1 -- 172.21.15.39:0/223753308 shutdown_connections 2023-06-22T17:26:16.319 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:16.316+0000 7f04b1ba2700 1 -- 172.21.15.39:0/223753308 wait complete. 2023-06-22T17:26:16.320 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 7 2023-06-22T17:26:16.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:16 smithi101 ceph-mon[85507]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:16.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:16 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:16.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:16 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:16.628 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:16 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:16.628 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:16 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:16.628 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:16 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:16.629 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:16 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:16.629 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:16 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/223753308' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:16.629 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:16 smithi101 ceph-mon[85507]: mds.? [v2:172.21.15.133:6836/967175183,v1:172.21.15.133:6837/967175183] up:boot 2023-06-22T17:26:16.629 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:16 smithi101 ceph-mon[85507]: fsmap 7 up:standby 2023-06-22T17:26:16.630 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:16 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-06-22T17:26:16.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:16 smithi039 ceph-mon[84805]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:16.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:16 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:16.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:16 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:16.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:16 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:16.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:16 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:16.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:16 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:16.685 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:16 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:16.685 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:16 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/223753308' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:16.685 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:16 smithi039 ceph-mon[84805]: mds.? [v2:172.21.15.133:6836/967175183,v1:172.21.15.133:6837/967175183] up:boot 2023-06-22T17:26:16.685 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:16 smithi039 ceph-mon[84805]: fsmap 7 up:standby 2023-06-22T17:26:16.685 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:16 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-06-22T17:26:16.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:16 smithi133 ceph-mon[86268]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:16.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:16 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:16.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:16 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:16.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:16 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:16.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:16 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:16.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:16 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:16.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:16 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:16.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:16 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/223753308' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:16.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:16 smithi133 ceph-mon[86268]: mds.? [v2:172.21.15.133:6836/967175183,v1:172.21.15.133:6837/967175183] up:boot 2023-06-22T17:26:16.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:16 smithi133 ceph-mon[86268]: fsmap 7 up:standby 2023-06-22T17:26:16.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:16 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-06-22T17:26:17.377 INFO:journalctl@ceph.mds.h.smithi101.stdout:Jun 22 17:26:17 smithi101 systemd[1]: Starting Ceph mds.h for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:26:17.839 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:17 smithi101 ceph-mon[85507]: Deploying daemon mds.h on smithi101 2023-06-22T17:26:17.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:17 smithi133 ceph-mon[86268]: Deploying daemon mds.h on smithi101 2023-06-22T17:26:17.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:17 smithi039 ceph-mon[84805]: Deploying daemon mds.h on smithi101 2023-06-22T17:26:18.126 INFO:journalctl@ceph.mds.h.smithi101.stdout:Jun 22 17:26:17 smithi101 ceph-mds[117629]: -- 172.21.15.101:0/1927914618 <== mon.2 v2:172.21.15.101:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e3e160a9c0 con 0x55e3e15e0800 2023-06-22T17:26:18.127 INFO:journalctl@ceph.mds.h.smithi101.stdout:Jun 22 17:26:17 smithi101 conmon[117605]: starting mds.h at 2023-06-22T17:26:18.127 INFO:journalctl@ceph.mds.h.smithi101.stdout:Jun 22 17:26:17 smithi101 bash[117565]: 6b62d292a6a743e194ca3932044028cf996afcad4e4d904662750565f345785d 2023-06-22T17:26:18.127 INFO:journalctl@ceph.mds.h.smithi101.stdout:Jun 22 17:26:17 smithi101 systemd[1]: Started Ceph mds.h for d88002e2-1120-11ee-9b2b-001a4aab830c. 2023-06-22T17:26:18.331 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T17:26:18.466 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.463+0000 7fb43689b700 1 -- 172.21.15.39:0/1844523880 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4300fcb20 msgr2=0x7fb4300fcf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:18.466 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.463+0000 7fb43689b700 1 --2- 172.21.15.39:0/1844523880 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4300fcb20 0x7fb4300fcf90 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb4240049f0 tx=0x7fb424009e30 comp rx=0 tx=0).stop 2023-06-22T17:26:18.466 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.463+0000 7fb43689b700 1 -- 172.21.15.39:0/1844523880 shutdown_connections 2023-06-22T17:26:18.466 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.463+0000 7fb43689b700 1 --2- 172.21.15.39:0/1844523880 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb4300fd6c0 0x7fb4301041e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:18.466 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.463+0000 7fb43689b700 1 --2- 172.21.15.39:0/1844523880 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb4300fc1d0 0x7fb4300fc5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:18.466 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.463+0000 7fb43689b700 1 --2- 172.21.15.39:0/1844523880 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4300fcb20 0x7fb4300fcf90 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:18.467 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.463+0000 7fb43689b700 1 -- 172.21.15.39:0/1844523880 >> 172.21.15.39:0/1844523880 conn(0x7fb4300f6a80 msgr2=0x7fb4300f8ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:18.467 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.463+0000 7fb43689b700 1 -- 172.21.15.39:0/1844523880 shutdown_connections 2023-06-22T17:26:18.467 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.464+0000 7fb43689b700 1 -- 172.21.15.39:0/1844523880 wait complete. 2023-06-22T17:26:18.467 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.464+0000 7fb43689b700 1 Processor -- start 2023-06-22T17:26:18.469 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.464+0000 7fb43689b700 1 -- start start 2023-06-22T17:26:18.469 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.464+0000 7fb43689b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4300fc1d0 0x7fb43010c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:18.469 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.464+0000 7fb43689b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb4300fcb20 0x7fb43010ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:18.469 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.464+0000 7fb43689b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb4300fd6c0 0x7fb430113f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:18.469 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.464+0000 7fb43689b700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb430114550 con 0x7fb4300fc1d0 2023-06-22T17:26:18.469 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.464+0000 7fb43689b700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4301146c0 con 0x7fb4300fcb20 2023-06-22T17:26:18.470 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.464+0000 7fb43689b700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb430114880 con 0x7fb4300fd6c0 2023-06-22T17:26:18.470 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.464+0000 7fb42ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4300fc1d0 0x7fb43010c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:18.470 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.465+0000 7fb42ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4300fc1d0 0x7fb43010c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58996/0 (socket says 172.21.15.39:58996) 2023-06-22T17:26:18.470 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.465+0000 7fb42ffff700 1 -- 172.21.15.39:0/2384138232 learned_addr learned my addr 172.21.15.39:0/2384138232 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:26:18.470 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.465+0000 7fb42f7fe700 1 --2- 172.21.15.39:0/2384138232 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb4300fcb20 0x7fb43010ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:18.470 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.465+0000 7fb434e38700 1 --2- 172.21.15.39:0/2384138232 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb4300fd6c0 0x7fb430113f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:18.470 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.465+0000 7fb42ffff700 1 -- 172.21.15.39:0/2384138232 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb4300fd6c0 msgr2=0x7fb430113f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:18.471 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.465+0000 7fb42ffff700 1 --2- 172.21.15.39:0/2384138232 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb4300fd6c0 0x7fb430113f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:18.471 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.465+0000 7fb42ffff700 1 -- 172.21.15.39:0/2384138232 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb4300fcb20 msgr2=0x7fb43010ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:18.471 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.465+0000 7fb42ffff700 1 --2- 172.21.15.39:0/2384138232 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb4300fcb20 0x7fb43010ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:18.471 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.465+0000 7fb42ffff700 1 -- 172.21.15.39:0/2384138232 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb424005040 con 0x7fb4300fc1d0 2023-06-22T17:26:18.471 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.465+0000 7fb434e38700 1 --2- 172.21.15.39:0/2384138232 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb4300fd6c0 0x7fb430113f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T17:26:18.471 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.465+0000 7fb42ffff700 1 --2- 172.21.15.39:0/2384138232 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4300fc1d0 0x7fb43010c970 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb420010e00 tx=0x7fb42000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:18.471 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.465+0000 7fb42cff9700 1 -- 172.21.15.39:0/2384138232 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42000eb50 con 0x7fb4300fc1d0 2023-06-22T17:26:18.471 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.465+0000 7fb42cff9700 1 -- 172.21.15.39:0/2384138232 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7fb42000ecb0 con 0x7fb4300fc1d0 2023-06-22T17:26:18.472 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.465+0000 7fb43689b700 1 -- 172.21.15.39:0/2384138232 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4300fae70 con 0x7fb4300fc1d0 2023-06-22T17:26:18.472 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.465+0000 7fb42cff9700 1 -- 172.21.15.39:0/2384138232 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb420012430 con 0x7fb4300fc1d0 2023-06-22T17:26:18.472 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.465+0000 7fb43689b700 1 -- 172.21.15.39:0/2384138232 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4300fb3e0 con 0x7fb4300fc1d0 2023-06-22T17:26:18.472 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.466+0000 7fb43689b700 1 -- 172.21.15.39:0/2384138232 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb430107590 con 0x7fb4300fc1d0 2023-06-22T17:26:18.473 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.468+0000 7fb42cff9700 1 -- 172.21.15.39:0/2384138232 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7fb420018030 con 0x7fb4300fc1d0 2023-06-22T17:26:18.473 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.468+0000 7fb42cff9700 1 --2- 172.21.15.39:0/2384138232 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fb41809b680 0x7fb41809db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:18.473 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.468+0000 7fb42f7fe700 1 --2- 172.21.15.39:0/2384138232 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fb41809b680 0x7fb41809db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:18.473 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.468+0000 7fb42cff9700 1 -- 172.21.15.39:0/2384138232 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fb420016070 con 0x7fb4300fc1d0 2023-06-22T17:26:18.473 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.469+0000 7fb42f7fe700 1 --2- 172.21.15.39:0/2384138232 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fb41809b680 0x7fb41809db30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb424009f00 tx=0x7fb424009e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:18.474 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.471+0000 7fb42cff9700 1 -- 172.21.15.39:0/2384138232 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fb4200930c0 con 0x7fb4300fc1d0 2023-06-22T17:26:18.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:18 smithi039 ceph-mon[84805]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:18.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:18 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:18.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:18 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:18.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:18 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:18.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:18 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:18.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:18 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:18.685 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:18 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:18.717 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.715+0000 7fb43689b700 1 -- 172.21.15.39:0/2384138232 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4300fb6c0 con 0x7fb4300fc1d0 2023-06-22T17:26:18.717 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.716+0000 7fb42cff9700 1 -- 172.21.15.39:0/2384138232 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6568 (secure 0 0 0) 0x7fb4200966e0 con 0x7fb4300fc1d0 2023-06-22T17:26:18.718 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:26:18.718 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3630756954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3630756954},{"type":"v1","addr":"172.21.15.39:6835","nonce":3630756954}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/768662129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":768662129},{"type":"v1","addr":"172.21.15.39:6837","nonce":768662129}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6833/3120964256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":3120964256},{"type":"v1","addr":"172.21.15.101:6833","nonce":3120964256}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/1691506923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":1691506923},{"type":"v1","addr":"172.21.15.133:6833","nonce":1691506923}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6835/4021581510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":4021581510},{"type":"v1","addr":"172.21.15.101:6835","nonce":4021581510}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6837/1721839744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6836","nonce":1721839744},{"type":"v1","addr":"172.21.15.101:6837","nonce":1721839744}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/1920373312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":1920373312},{"type":"v1","addr":"172.21.15.133:6835","nonce":1920373312}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/967175183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":967175183},{"type":"v1","addr":"172.21.15.133:6837","nonce":967175183}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2023-06-22T17:26:18.720 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.718+0000 7fb43689b700 1 -- 172.21.15.39:0/2384138232 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fb41809b680 msgr2=0x7fb41809db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:18.720 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.718+0000 7fb43689b700 1 --2- 172.21.15.39:0/2384138232 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fb41809b680 0x7fb41809db30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb424009f00 tx=0x7fb424009e30 comp rx=0 tx=0).stop 2023-06-22T17:26:18.720 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.718+0000 7fb43689b700 1 -- 172.21.15.39:0/2384138232 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4300fc1d0 msgr2=0x7fb43010c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:18.720 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.718+0000 7fb43689b700 1 --2- 172.21.15.39:0/2384138232 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4300fc1d0 0x7fb43010c970 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb420010e00 tx=0x7fb42000b040 comp rx=0 tx=0).stop 2023-06-22T17:26:18.720 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.718+0000 7fb43689b700 1 -- 172.21.15.39:0/2384138232 shutdown_connections 2023-06-22T17:26:18.720 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.718+0000 7fb43689b700 1 --2- 172.21.15.39:0/2384138232 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fb41809b680 0x7fb41809db30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:18.721 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.718+0000 7fb43689b700 1 --2- 172.21.15.39:0/2384138232 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb4300fd6c0 0x7fb430113f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:18.721 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.718+0000 7fb43689b700 1 --2- 172.21.15.39:0/2384138232 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4300fc1d0 0x7fb43010c970 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:18.721 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.718+0000 7fb43689b700 1 --2- 172.21.15.39:0/2384138232 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb4300fcb20 0x7fb43010ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:18.721 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.718+0000 7fb43689b700 1 -- 172.21.15.39:0/2384138232 >> 172.21.15.39:0/2384138232 conn(0x7fb4300f6a80 msgr2=0x7fb4300f8c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:18.721 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.718+0000 7fb43689b700 1 -- 172.21.15.39:0/2384138232 shutdown_connections 2023-06-22T17:26:18.721 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:18.718+0000 7fb43689b700 1 -- 172.21.15.39:0/2384138232 wait complete. 2023-06-22T17:26:18.722 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 9 2023-06-22T17:26:18.730 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:18 smithi133 ceph-mon[86268]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:18.730 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:18 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:18.730 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:18 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:18.730 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:18 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:18.730 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:18 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:18.731 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:18 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:18.731 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:18 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:18.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:18 smithi101 ceph-mon[85507]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:18.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:18 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:18.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:18 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:18.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:18 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:18.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:18 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:18.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:18 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:18.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:18 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:19.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:19 smithi101 ceph-mon[85507]: Deploying daemon mds.i on smithi133 2023-06-22T17:26:19.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:19 smithi101 ceph-mon[85507]: mds.? [v2:172.21.15.101:6836/1721839744,v1:172.21.15.101:6837/1721839744] up:boot 2023-06-22T17:26:19.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:19 smithi101 ceph-mon[85507]: fsmap 8 up:standby 2023-06-22T17:26:19.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:19 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-06-22T17:26:19.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:19 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/2384138232' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:19.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:19 smithi133 ceph-mon[86268]: Deploying daemon mds.i on smithi133 2023-06-22T17:26:19.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:19 smithi133 ceph-mon[86268]: mds.? [v2:172.21.15.101:6836/1721839744,v1:172.21.15.101:6837/1721839744] up:boot 2023-06-22T17:26:19.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:19 smithi133 ceph-mon[86268]: fsmap 8 up:standby 2023-06-22T17:26:19.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:19 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-06-22T17:26:19.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:19 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/2384138232' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:19.922 INFO:journalctl@ceph.mds.i.smithi133.stdout:Jun 22 17:26:19 smithi133 systemd[1]: Starting Ceph mds.i for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:26:19.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:19 smithi039 ceph-mon[84805]: Deploying daemon mds.i on smithi133 2023-06-22T17:26:19.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:19 smithi039 ceph-mon[84805]: mds.? [v2:172.21.15.101:6836/1721839744,v1:172.21.15.101:6837/1721839744] up:boot 2023-06-22T17:26:19.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:19 smithi039 ceph-mon[84805]: fsmap 8 up:standby 2023-06-22T17:26:19.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:19 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-06-22T17:26:19.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:19 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/2384138232' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:20.492 INFO:journalctl@ceph.mds.i.smithi133.stdout:Jun 22 17:26:20 smithi133 conmon[120573]: starting mds.i at 2023-06-22T17:26:20.492 INFO:journalctl@ceph.mds.i.smithi133.stdout:Jun 22 17:26:20 smithi133 bash[120533]: 952bd5014ea240c97ae4e2bcfac9d89bcf179756a46242214ffa3cf6d783e3cc 2023-06-22T17:26:20.492 INFO:journalctl@ceph.mds.i.smithi133.stdout:Jun 22 17:26:20 smithi133 systemd[1]: Started Ceph mds.i for d88002e2-1120-11ee-9b2b-001a4aab830c. 2023-06-22T17:26:20.732 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T17:26:20.757 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:20 smithi039 ceph-mon[84805]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:20.757 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:20 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:20.757 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:20 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:20.757 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:20 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:20.757 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:20 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:20.757 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:20 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:20.758 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:20 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:20.869 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.866+0000 7f6d0c83f700 1 -- 172.21.15.39:0/3949157012 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d080fce20 msgr2=0x7f6d080fd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:20.869 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.866+0000 7f6d0c83f700 1 --2- 172.21.15.39:0/3949157012 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d080fce20 0x7f6d080fd270 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6cf00049c0 tx=0x7f6cf0009e30 comp rx=0 tx=0).stop 2023-06-22T17:26:20.869 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.866+0000 7f6d0c83f700 1 -- 172.21.15.39:0/3949157012 shutdown_connections 2023-06-22T17:26:20.869 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.866+0000 7f6d0c83f700 1 --2- 172.21.15.39:0/3949157012 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6d080fd7b0 0x7f6d08104070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:20.870 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.866+0000 7f6d0c83f700 1 --2- 172.21.15.39:0/3949157012 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f6d080fbc10 0x7f6d080fc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:20.870 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.866+0000 7f6d0c83f700 1 --2- 172.21.15.39:0/3949157012 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d080fce20 0x7f6d080fd270 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:20.870 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.866+0000 7f6d0c83f700 1 -- 172.21.15.39:0/3949157012 >> 172.21.15.39:0/3949157012 conn(0x7f6d080f7350 msgr2=0x7f6d080f97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:20.870 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.866+0000 7f6d0c83f700 1 -- 172.21.15.39:0/3949157012 shutdown_connections 2023-06-22T17:26:20.870 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.866+0000 7f6d0c83f700 1 -- 172.21.15.39:0/3949157012 wait complete. 2023-06-22T17:26:20.870 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.867+0000 7f6d0c83f700 1 Processor -- start 2023-06-22T17:26:20.870 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.867+0000 7f6d0c83f700 1 -- start start 2023-06-22T17:26:20.872 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.867+0000 7f6d0c83f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6d080fbc10 0x7f6d080666c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:20.873 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.867+0000 7f6d0c83f700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d080fce20 0x7f6d081127a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:20.873 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.867+0000 7f6d0c83f700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f6d080fd7b0 0x7f6d08114dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:20.873 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.867+0000 7f6d0c83f700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d081155b0 con 0x7f6d080fce20 2023-06-22T17:26:20.873 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.867+0000 7f6d0c83f700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d08115720 con 0x7f6d080fd7b0 2023-06-22T17:26:20.873 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.867+0000 7f6d0c83f700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d081158e0 con 0x7f6d080fbc10 2023-06-22T17:26:20.873 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.867+0000 7f6cfdd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d080fce20 0x7f6d081127a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:20.874 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.867+0000 7f6cfdd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d080fce20 0x7f6d081127a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59016/0 (socket says 172.21.15.39:59016) 2023-06-22T17:26:20.876 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.867+0000 7f6cfdd9b700 1 -- 172.21.15.39:0/1970107447 learned_addr learned my addr 172.21.15.39:0/1970107447 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:26:20.876 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.867+0000 7f6d0659c700 1 --2- 172.21.15.39:0/1970107447 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6d080fbc10 0x7f6d080666c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:20.877 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.868+0000 7f6d06d9d700 1 --2- 172.21.15.39:0/1970107447 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f6d080fd7b0 0x7f6d08114dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:20.877 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.868+0000 7f6cfdd9b700 1 -- 172.21.15.39:0/1970107447 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6d080fbc10 msgr2=0x7f6d080666c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:20.877 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.868+0000 7f6cfdd9b700 1 --2- 172.21.15.39:0/1970107447 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6d080fbc10 0x7f6d080666c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:20.877 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.868+0000 7f6cfdd9b700 1 -- 172.21.15.39:0/1970107447 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f6d080fd7b0 msgr2=0x7f6d08114dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:20.877 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.868+0000 7f6cfdd9b700 1 --2- 172.21.15.39:0/1970107447 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f6d080fd7b0 0x7f6d08114dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:20.877 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.868+0000 7f6cfdd9b700 1 -- 172.21.15.39:0/1970107447 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf0005040 con 0x7f6d080fce20 2023-06-22T17:26:20.877 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.868+0000 7f6d06d9d700 1 --2- 172.21.15.39:0/1970107447 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f6d080fd7b0 0x7f6d08114dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-22T17:26:20.878 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.868+0000 7f6cfdd9b700 1 --2- 172.21.15.39:0/1970107447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d080fce20 0x7f6d081127a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6cf0009f00 tx=0x7f6cf0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:20.878 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.868+0000 7f6cff7fe700 1 -- 172.21.15.39:0/1970107447 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf000e040 con 0x7f6d080fce20 2023-06-22T17:26:20.878 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.868+0000 7f6cff7fe700 1 -- 172.21.15.39:0/1970107447 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f6cf0013420 con 0x7f6d080fce20 2023-06-22T17:26:20.878 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.868+0000 7f6d0c83f700 1 -- 172.21.15.39:0/1970107447 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d08115b00 con 0x7f6d080fce20 2023-06-22T17:26:20.878 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.868+0000 7f6cff7fe700 1 -- 172.21.15.39:0/1970107447 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf001c620 con 0x7f6d080fce20 2023-06-22T17:26:20.879 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.868+0000 7f6d0c83f700 1 -- 172.21.15.39:0/1970107447 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d08115fe0 con 0x7f6d080fce20 2023-06-22T17:26:20.879 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.869+0000 7f6d0c83f700 1 -- 172.21.15.39:0/1970107447 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d08107540 con 0x7f6d080fce20 2023-06-22T17:26:20.879 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.870+0000 7f6cff7fe700 1 -- 172.21.15.39:0/1970107447 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f6cf0013590 con 0x7f6d080fce20 2023-06-22T17:26:20.880 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.870+0000 7f6cff7fe700 1 --2- 172.21.15.39:0/1970107447 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f6cf409b680 0x7f6cf409db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:20.880 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.871+0000 7f6d0659c700 1 --2- 172.21.15.39:0/1970107447 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f6cf409b680 0x7f6cf409db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:20.880 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.871+0000 7f6cff7fe700 1 -- 172.21.15.39:0/1970107447 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f6cf00c65a0 con 0x7f6d080fce20 2023-06-22T17:26:20.881 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:20 smithi101 ceph-mon[85507]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:20.881 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:20 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:20.881 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:20 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:20.882 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:20 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:20.882 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:20 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:20.882 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:20 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:20.882 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:20 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:20.883 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.871+0000 7f6d0659c700 1 --2- 172.21.15.39:0/1970107447 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f6cf409b680 0x7f6cf409db30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6d080fcc80 tx=0x7f6cf8009040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:20.883 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:20.874+0000 7f6cff7fe700 1 -- 172.21.15.39:0/1970107447 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f6cf00922a0 con 0x7f6d080fce20 2023-06-22T17:26:20.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:20 smithi133 ceph-mon[86268]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:20.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:20 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:20.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:20 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:20.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:20 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:20.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:20 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:20.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:20 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:20.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:20 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:21.127 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:21.125+0000 7f6d0c83f700 1 -- 172.21.15.39:0/1970107447 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d081161d0 con 0x7f6d080fce20 2023-06-22T17:26:21.127 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:21.125+0000 7f6cff7fe700 1 -- 172.21.15.39:0/1970107447 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7330 (secure 0 0 0) 0x7f6cf00958c0 con 0x7f6d080fce20 2023-06-22T17:26:21.128 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:26:21.128 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3630756954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3630756954},{"type":"v1","addr":"172.21.15.39:6835","nonce":3630756954}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/768662129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":768662129},{"type":"v1","addr":"172.21.15.39:6837","nonce":768662129}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6833/3120964256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":3120964256},{"type":"v1","addr":"172.21.15.101:6833","nonce":3120964256}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/1691506923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":1691506923},{"type":"v1","addr":"172.21.15.133:6833","nonce":1691506923}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6835/4021581510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":4021581510},{"type":"v1","addr":"172.21.15.101:6835","nonce":4021581510}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6837/1721839744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6836","nonce":1721839744},{"type":"v1","addr":"172.21.15.101:6837","nonce":1721839744}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/1920373312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":1920373312},{"type":"v1","addr":"172.21.15.133:6835","nonce":1920373312}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/967175183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":967175183},{"type":"v1","addr":"172.21.15.133:6837","nonce":967175183}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24523,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6839/1006809426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1006809426},{"type":"v1","addr":"172.21.15.133:6839","nonce":1006809426}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-06-22T17:26:21.130 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:21.128+0000 7f6d0c83f700 1 -- 172.21.15.39:0/1970107447 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f6cf409b680 msgr2=0x7f6cf409db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:21.130 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:21.128+0000 7f6d0c83f700 1 --2- 172.21.15.39:0/1970107447 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f6cf409b680 0x7f6cf409db30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6d080fcc80 tx=0x7f6cf8009040 comp rx=0 tx=0).stop 2023-06-22T17:26:21.130 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:21.128+0000 7f6d0c83f700 1 -- 172.21.15.39:0/1970107447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d080fce20 msgr2=0x7f6d081127a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:21.130 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:21.128+0000 7f6d0c83f700 1 --2- 172.21.15.39:0/1970107447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d080fce20 0x7f6d081127a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6cf0009f00 tx=0x7f6cf0009e30 comp rx=0 tx=0).stop 2023-06-22T17:26:21.130 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:21.128+0000 7f6d0c83f700 1 -- 172.21.15.39:0/1970107447 shutdown_connections 2023-06-22T17:26:21.130 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:21.128+0000 7f6d0c83f700 1 --2- 172.21.15.39:0/1970107447 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f6cf409b680 0x7f6cf409db30 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:21.131 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:21.128+0000 7f6d0c83f700 1 --2- 172.21.15.39:0/1970107447 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f6d080fd7b0 0x7f6d08114dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:21.131 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:21.128+0000 7f6d0c83f700 1 --2- 172.21.15.39:0/1970107447 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6d080fbc10 0x7f6d080666c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:21.131 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:21.128+0000 7f6d0c83f700 1 --2- 172.21.15.39:0/1970107447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d080fce20 0x7f6d081127a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:21.131 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:21.128+0000 7f6d0c83f700 1 -- 172.21.15.39:0/1970107447 >> 172.21.15.39:0/1970107447 conn(0x7f6d080f7350 msgr2=0x7f6d080f9440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:21.131 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:21.128+0000 7f6d0c83f700 1 -- 172.21.15.39:0/1970107447 shutdown_connections 2023-06-22T17:26:21.131 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:21.128+0000 7f6d0c83f700 1 -- 172.21.15.39:0/1970107447 wait complete. 2023-06-22T17:26:21.132 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 10 2023-06-22T17:26:21.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:21 smithi039 ceph-mon[84805]: Deploying daemon mds.g on smithi039 2023-06-22T17:26:21.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:21 smithi039 ceph-mon[84805]: mds.? [v2:172.21.15.133:6838/1006809426,v1:172.21.15.133:6839/1006809426] up:boot 2023-06-22T17:26:21.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:21 smithi039 ceph-mon[84805]: fsmap 9 up:standby 2023-06-22T17:26:21.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:21 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-06-22T17:26:21.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:21 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/1970107447' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:21.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:21 smithi101 ceph-mon[85507]: Deploying daemon mds.g on smithi039 2023-06-22T17:26:21.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:21 smithi101 ceph-mon[85507]: mds.? [v2:172.21.15.133:6838/1006809426,v1:172.21.15.133:6839/1006809426] up:boot 2023-06-22T17:26:21.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:21 smithi101 ceph-mon[85507]: fsmap 9 up:standby 2023-06-22T17:26:21.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:21 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-06-22T17:26:21.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:21 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/1970107447' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:21.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:21 smithi133 ceph-mon[86268]: Deploying daemon mds.g on smithi039 2023-06-22T17:26:21.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:21 smithi133 ceph-mon[86268]: mds.? [v2:172.21.15.133:6838/1006809426,v1:172.21.15.133:6839/1006809426] up:boot 2023-06-22T17:26:21.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:21 smithi133 ceph-mon[86268]: fsmap 9 up:standby 2023-06-22T17:26:21.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:21 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-06-22T17:26:21.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:21 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/1970107447' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:22.184 INFO:journalctl@ceph.mds.g.smithi039.stdout:Jun 22 17:26:21 smithi039 systemd[1]: Starting Ceph mds.g for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:26:22.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:22 smithi039 ceph-mon[84805]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:22.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:22 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:22.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:22 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:22.684 INFO:journalctl@ceph.mds.g.smithi039.stdout:Jun 22 17:26:22 smithi039 ceph-mds[129456]: -- 172.21.15.39:0/3162374233 <== mon.0 v2:172.21.15.39:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562b030fe9c0 con 0x562b030d5000 2023-06-22T17:26:22.684 INFO:journalctl@ceph.mds.g.smithi039.stdout:Jun 22 17:26:22 smithi039 conmon[129431]: starting mds.g at 2023-06-22T17:26:22.684 INFO:journalctl@ceph.mds.g.smithi039.stdout:Jun 22 17:26:22 smithi039 bash[129391]: c9d181fe2adab7ddbe9c9071dd1b36164a03b74322acd90a9bded50dcab5d5e8 2023-06-22T17:26:22.685 INFO:journalctl@ceph.mds.g.smithi039.stdout:Jun 22 17:26:22 smithi039 systemd[1]: Started Ceph mds.g for d88002e2-1120-11ee-9b2b-001a4aab830c. 2023-06-22T17:26:22.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:22 smithi101 ceph-mon[85507]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:22.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:22 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:22.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:22 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:22.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:22 smithi133 ceph-mon[86268]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:22.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:22 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:22.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:22 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:23.143 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T17:26:23.273 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.270+0000 7f177b2fb700 1 -- 172.21.15.39:0/108582352 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17740fce20 msgr2=0x7f17740fd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:23.273 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.270+0000 7f177b2fb700 1 --2- 172.21.15.39:0/108582352 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17740fce20 0x7f17740fd270 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f17700049c0 tx=0x7f1770009e30 comp rx=0 tx=0).stop 2023-06-22T17:26:23.273 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.270+0000 7f177b2fb700 1 -- 172.21.15.39:0/108582352 shutdown_connections 2023-06-22T17:26:23.273 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.270+0000 7f177b2fb700 1 --2- 172.21.15.39:0/108582352 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17740fd7b0 0x7f1774104070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:23.273 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.271+0000 7f177b2fb700 1 --2- 172.21.15.39:0/108582352 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f17740fbc10 0x7f17740fc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:23.273 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.271+0000 7f177b2fb700 1 --2- 172.21.15.39:0/108582352 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17740fce20 0x7f17740fd270 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:23.274 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.271+0000 7f177b2fb700 1 -- 172.21.15.39:0/108582352 >> 172.21.15.39:0/108582352 conn(0x7f17740f7350 msgr2=0x7f17740f97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:23.274 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.271+0000 7f177b2fb700 1 -- 172.21.15.39:0/108582352 shutdown_connections 2023-06-22T17:26:23.274 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.271+0000 7f177b2fb700 1 -- 172.21.15.39:0/108582352 wait complete. 2023-06-22T17:26:23.275 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.271+0000 7f177b2fb700 1 Processor -- start 2023-06-22T17:26:23.276 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.271+0000 7f177b2fb700 1 -- start start 2023-06-22T17:26:23.276 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.271+0000 7f177b2fb700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f17740fbc10 0x7f177410d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:23.276 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.271+0000 7f177b2fb700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17740fce20 0x7f177410dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:23.276 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.271+0000 7f177b2fb700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17740fd7b0 0x7f1774114f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:23.276 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.271+0000 7f177b2fb700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1774115580 con 0x7f17740fd7b0 2023-06-22T17:26:23.276 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.271+0000 7f177b2fb700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17741156f0 con 0x7f17740fbc10 2023-06-22T17:26:23.276 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.271+0000 7f177b2fb700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17741158b0 con 0x7f17740fce20 2023-06-22T17:26:23.277 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.271+0000 7f1779898700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17740fd7b0 0x7f1774114f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:23.277 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.271+0000 7f1779898700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17740fd7b0 0x7f1774114f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59044/0 (socket says 172.21.15.39:59044) 2023-06-22T17:26:23.277 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.271+0000 7f1779898700 1 -- 172.21.15.39:0/2673156129 learned_addr learned my addr 172.21.15.39:0/2673156129 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:26:23.277 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.272+0000 7f1779097700 1 --2- 172.21.15.39:0/2673156129 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f17740fbc10 0x7f177410d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:23.277 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.272+0000 7f1778896700 1 --2- 172.21.15.39:0/2673156129 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17740fce20 0x7f177410dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:23.277 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.272+0000 7f1779898700 1 -- 172.21.15.39:0/2673156129 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17740fce20 msgr2=0x7f177410dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:23.277 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.272+0000 7f1779898700 1 --2- 172.21.15.39:0/2673156129 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17740fce20 0x7f177410dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:23.278 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.272+0000 7f1779898700 1 -- 172.21.15.39:0/2673156129 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f17740fbc10 msgr2=0x7f177410d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:23.278 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.272+0000 7f1779898700 1 --2- 172.21.15.39:0/2673156129 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f17740fbc10 0x7f177410d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:23.278 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.272+0000 7f1779898700 1 -- 172.21.15.39:0/2673156129 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1770005040 con 0x7f17740fd7b0 2023-06-22T17:26:23.278 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.272+0000 7f1779898700 1 --2- 172.21.15.39:0/2673156129 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17740fd7b0 0x7f1774114f70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f176c001d00 tx=0x7f176c009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:23.278 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.272+0000 7f1769ffb700 1 -- 172.21.15.39:0/2673156129 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f176c001e60 con 0x7f17740fd7b0 2023-06-22T17:26:23.278 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.272+0000 7f1769ffb700 1 -- 172.21.15.39:0/2673156129 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f176c00f040 con 0x7f17740fd7b0 2023-06-22T17:26:23.278 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.272+0000 7f1769ffb700 1 -- 172.21.15.39:0/2673156129 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f176c0090d0 con 0x7f17740fd7b0 2023-06-22T17:26:23.278 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.272+0000 7f177b2fb700 1 -- 172.21.15.39:0/2673156129 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1774115b90 con 0x7f17740fd7b0 2023-06-22T17:26:23.279 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.272+0000 7f177b2fb700 1 -- 172.21.15.39:0/2673156129 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1774116080 con 0x7f17740fd7b0 2023-06-22T17:26:23.279 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.273+0000 7f177b2fb700 1 -- 172.21.15.39:0/2673156129 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1774107570 con 0x7f17740fd7b0 2023-06-22T17:26:23.279 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.274+0000 7f1769ffb700 1 -- 172.21.15.39:0/2673156129 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f176c009500 con 0x7f17740fd7b0 2023-06-22T17:26:23.280 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.275+0000 7f1769ffb700 1 --2- 172.21.15.39:0/2673156129 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f176009b700 0x7f176009dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:23.280 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.275+0000 7f1779097700 1 --2- 172.21.15.39:0/2673156129 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f176009b700 0x7f176009dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:23.280 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.275+0000 7f1769ffb700 1 -- 172.21.15.39:0/2673156129 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f176c0c46b0 con 0x7f17740fd7b0 2023-06-22T17:26:23.280 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.275+0000 7f1779097700 1 --2- 172.21.15.39:0/2673156129 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f176009b700 0x7f176009dbb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1764009fd0 tx=0x7f176400b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:23.281 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.279+0000 7f1769ffb700 1 -- 172.21.15.39:0/2673156129 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f176c090440 con 0x7f17740fd7b0 2023-06-22T17:26:23.527 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.525+0000 7f177b2fb700 1 -- 172.21.15.39:0/2673156129 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1774116450 con 0x7f17740fd7b0 2023-06-22T17:26:23.527 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.525+0000 7f1769ffb700 1 -- 172.21.15.39:0/2673156129 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8088 (secure 0 0 0) 0x7f176c093a60 con 0x7f17740fd7b0 2023-06-22T17:26:23.527 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:26:23.528 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3630756954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3630756954},{"type":"v1","addr":"172.21.15.39:6835","nonce":3630756954}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/768662129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":768662129},{"type":"v1","addr":"172.21.15.39:6837","nonce":768662129}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/3636987214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":3636987214},{"type":"v1","addr":"172.21.15.39:6839","nonce":3636987214}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6833/3120964256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":3120964256},{"type":"v1","addr":"172.21.15.101:6833","nonce":3120964256}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/1691506923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":1691506923},{"type":"v1","addr":"172.21.15.133:6833","nonce":1691506923}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6835/4021581510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":4021581510},{"type":"v1","addr":"172.21.15.101:6835","nonce":4021581510}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6837/1721839744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6836","nonce":1721839744},{"type":"v1","addr":"172.21.15.101:6837","nonce":1721839744}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/1920373312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":1920373312},{"type":"v1","addr":"172.21.15.133:6835","nonce":1920373312}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/967175183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":967175183},{"type":"v1","addr":"172.21.15.133:6837","nonce":967175183}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24523,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6839/1006809426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1006809426},{"type":"v1","addr":"172.21.15.133:6839","nonce":1006809426}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-06-22T17:26:23.530 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.528+0000 7f177b2fb700 1 -- 172.21.15.39:0/2673156129 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f176009b700 msgr2=0x7f176009dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:23.530 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.528+0000 7f177b2fb700 1 --2- 172.21.15.39:0/2673156129 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f176009b700 0x7f176009dbb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1764009fd0 tx=0x7f176400b040 comp rx=0 tx=0).stop 2023-06-22T17:26:23.530 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.528+0000 7f177b2fb700 1 -- 172.21.15.39:0/2673156129 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17740fd7b0 msgr2=0x7f1774114f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:23.530 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.528+0000 7f177b2fb700 1 --2- 172.21.15.39:0/2673156129 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17740fd7b0 0x7f1774114f70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f176c001d00 tx=0x7f176c009f30 comp rx=0 tx=0).stop 2023-06-22T17:26:23.530 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.528+0000 7f177b2fb700 1 -- 172.21.15.39:0/2673156129 shutdown_connections 2023-06-22T17:26:23.530 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.528+0000 7f177b2fb700 1 --2- 172.21.15.39:0/2673156129 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f176009b700 0x7f176009dbb0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:23.531 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.528+0000 7f177b2fb700 1 --2- 172.21.15.39:0/2673156129 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17740fd7b0 0x7f1774114f70 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:23.531 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.528+0000 7f177b2fb700 1 --2- 172.21.15.39:0/2673156129 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f17740fbc10 0x7f177410d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:23.531 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.528+0000 7f177b2fb700 1 --2- 172.21.15.39:0/2673156129 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17740fce20 0x7f177410dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:23.531 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.528+0000 7f177b2fb700 1 -- 172.21.15.39:0/2673156129 >> 172.21.15.39:0/2673156129 conn(0x7f17740f7350 msgr2=0x7f1774065c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:23.531 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.528+0000 7f177b2fb700 1 -- 172.21.15.39:0/2673156129 shutdown_connections 2023-06-22T17:26:23.531 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:23.528+0000 7f177b2fb700 1 -- 172.21.15.39:0/2673156129 wait complete. 2023-06-22T17:26:23.532 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 11 2023-06-22T17:26:23.864 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:23 smithi039 ceph-mon[84805]: mds.? [v2:172.21.15.39:6838/3636987214,v1:172.21.15.39:6839/3636987214] up:boot 2023-06-22T17:26:23.864 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:23 smithi039 ceph-mon[84805]: fsmap 10 up:standby 2023-06-22T17:26:23.864 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:23 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-06-22T17:26:23.864 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:23 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:23.864 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:23 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:23.865 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:23 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:23.865 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:23 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:23.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:23 smithi101 ceph-mon[85507]: mds.? [v2:172.21.15.39:6838/3636987214,v1:172.21.15.39:6839/3636987214] up:boot 2023-06-22T17:26:23.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:23 smithi101 ceph-mon[85507]: fsmap 10 up:standby 2023-06-22T17:26:23.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:23 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-06-22T17:26:23.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:23 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:23.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:23 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:23.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:23 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:23.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:23 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:23.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:23 smithi133 ceph-mon[86268]: mds.? [v2:172.21.15.39:6838/3636987214,v1:172.21.15.39:6839/3636987214] up:boot 2023-06-22T17:26:23.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:23 smithi133 ceph-mon[86268]: fsmap 10 up:standby 2023-06-22T17:26:23.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:23 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-06-22T17:26:23.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:23 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:23.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:23 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:23.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:23 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:23.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:23 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:24.183 INFO:journalctl@ceph.mds.a.smithi039.stdout:Jun 22 17:26:24 smithi039 systemd[1]: Starting Ceph mds.a for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:26:24.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:24 smithi101 ceph-mon[85507]: Deploying daemon mds.a on smithi039 2023-06-22T17:26:24.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:24 smithi101 ceph-mon[85507]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:24.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:24 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/2673156129' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:24.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:24 smithi133 ceph-mon[86268]: Deploying daemon mds.a on smithi039 2023-06-22T17:26:24.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:24 smithi133 ceph-mon[86268]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:24.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:24 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/2673156129' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:24.933 INFO:journalctl@ceph.mds.a.smithi039.stdout:Jun 22 17:26:24 smithi039 ceph-mds[130055]: -- 172.21.15.39:0/1774558789 <== mon.0 v2:172.21.15.39:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556a4b7c09c0 con 0x556a4b797000 2023-06-22T17:26:24.934 INFO:journalctl@ceph.mds.a.smithi039.stdout:Jun 22 17:26:24 smithi039 conmon[130031]: starting mds.a at 2023-06-22T17:26:24.934 INFO:journalctl@ceph.mds.a.smithi039.stdout:Jun 22 17:26:24 smithi039 bash[129990]: b9d7e00c4a5bad42c475d71f1ceb92e32b82ef36cf9783490c0f8ad7b64aa671 2023-06-22T17:26:24.934 INFO:journalctl@ceph.mds.a.smithi039.stdout:Jun 22 17:26:24 smithi039 systemd[1]: Started Ceph mds.a for d88002e2-1120-11ee-9b2b-001a4aab830c. 2023-06-22T17:26:24.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:24 smithi039 ceph-mon[84805]: Deploying daemon mds.a on smithi039 2023-06-22T17:26:24.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:24 smithi039 ceph-mon[84805]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:24.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:24 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/2673156129' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:25.543 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T17:26:25.682 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.678+0000 7f338826e700 1 -- 172.21.15.39:0/2337398088 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33800fd600 msgr2=0x7f33800fdff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:25.682 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.678+0000 7f338826e700 1 --2- 172.21.15.39:0/2337398088 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33800fd600 0x7f33800fdff0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f33740049c0 tx=0x7f3374009e30 comp rx=0 tx=0).stop 2023-06-22T17:26:25.682 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.678+0000 7f338826e700 1 -- 172.21.15.39:0/2337398088 shutdown_connections 2023-06-22T17:26:25.683 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.678+0000 7f338826e700 1 --2- 172.21.15.39:0/2337398088 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f33800fe720 0x7f3380105240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:25.683 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.678+0000 7f338826e700 1 --2- 172.21.15.39:0/2337398088 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33800fccb0 0x7f33800fd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:25.683 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.678+0000 7f338826e700 1 --2- 172.21.15.39:0/2337398088 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33800fd600 0x7f33800fdff0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:25.683 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.678+0000 7f338826e700 1 -- 172.21.15.39:0/2337398088 >> 172.21.15.39:0/2337398088 conn(0x7f33800f84a0 msgr2=0x7f33800fa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:25.683 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.678+0000 7f338826e700 1 -- 172.21.15.39:0/2337398088 shutdown_connections 2023-06-22T17:26:25.683 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.679+0000 7f338826e700 1 -- 172.21.15.39:0/2337398088 wait complete. 2023-06-22T17:26:25.683 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.679+0000 7f338826e700 1 Processor -- start 2023-06-22T17:26:25.684 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.679+0000 7f338826e700 1 -- start start 2023-06-22T17:26:25.684 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.679+0000 7f338826e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33800fccb0 0x7f338010da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:25.684 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.679+0000 7f338826e700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f33800fd600 0x7f338010df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:25.684 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.679+0000 7f338826e700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33800fe720 0x7f3380115010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:25.684 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.679+0000 7f338826e700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3380115620 con 0x7f33800fccb0 2023-06-22T17:26:25.684 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.679+0000 7f338826e700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3380115790 con 0x7f33800fd600 2023-06-22T17:26:25.685 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.679+0000 7f338826e700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3380115950 con 0x7f33800fe720 2023-06-22T17:26:25.685 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.679+0000 7f338600a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33800fccb0 0x7f338010da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:25.685 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.679+0000 7f338600a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33800fccb0 0x7f338010da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37222/0 (socket says 172.21.15.39:37222) 2023-06-22T17:26:25.685 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.679+0000 7f338600a700 1 -- 172.21.15.39:0/3871526725 learned_addr learned my addr 172.21.15.39:0/3871526725 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:26:25.685 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.679+0000 7f338680b700 1 --2- 172.21.15.39:0/3871526725 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33800fe720 0x7f3380115010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:25.685 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.679+0000 7f3385809700 1 --2- 172.21.15.39:0/3871526725 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f33800fd600 0x7f338010df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:25.685 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.680+0000 7f338600a700 1 -- 172.21.15.39:0/3871526725 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33800fe720 msgr2=0x7f3380115010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:25.686 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.680+0000 7f338600a700 1 --2- 172.21.15.39:0/3871526725 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33800fe720 0x7f3380115010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:25.686 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.680+0000 7f338600a700 1 -- 172.21.15.39:0/3871526725 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f33800fd600 msgr2=0x7f338010df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:25.686 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.680+0000 7f338600a700 1 --2- 172.21.15.39:0/3871526725 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f33800fd600 0x7f338010df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:25.686 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.680+0000 7f338600a700 1 -- 172.21.15.39:0/3871526725 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3374005040 con 0x7f33800fccb0 2023-06-22T17:26:25.686 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.680+0000 7f338680b700 1 --2- 172.21.15.39:0/3871526725 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33800fe720 0x7f3380115010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T17:26:25.686 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.680+0000 7f338600a700 1 --2- 172.21.15.39:0/3871526725 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33800fccb0 0x7f338010da40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3370001e00 tx=0x7f3370009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:25.686 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.680+0000 7f337effd700 1 -- 172.21.15.39:0/3871526725 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3370007b30 con 0x7f33800fccb0 2023-06-22T17:26:25.686 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.680+0000 7f338826e700 1 -- 172.21.15.39:0/3871526725 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3380115c30 con 0x7f33800fccb0 2023-06-22T17:26:25.687 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.680+0000 7f337effd700 1 -- 172.21.15.39:0/3871526725 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f337000ee40 con 0x7f33800fccb0 2023-06-22T17:26:25.687 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.680+0000 7f337effd700 1 -- 172.21.15.39:0/3871526725 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3370014400 con 0x7f33800fccb0 2023-06-22T17:26:25.688 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.680+0000 7f338826e700 1 -- 172.21.15.39:0/3871526725 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3380116120 con 0x7f33800fccb0 2023-06-22T17:26:25.688 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.681+0000 7f338826e700 1 -- 172.21.15.39:0/3871526725 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33801042b0 con 0x7f33800fccb0 2023-06-22T17:26:25.688 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.682+0000 7f337effd700 1 -- 172.21.15.39:0/3871526725 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f337000e960 con 0x7f33800fccb0 2023-06-22T17:26:25.688 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.682+0000 7f337effd700 1 --2- 172.21.15.39:0/3871526725 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f336809b680 0x7f336809db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:25.689 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.683+0000 7f337effd700 1 -- 172.21.15.39:0/3871526725 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f33700c4790 con 0x7f33800fccb0 2023-06-22T17:26:25.689 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.683+0000 7f3385809700 1 --2- 172.21.15.39:0/3871526725 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f336809b680 0x7f336809db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:25.689 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.683+0000 7f3385809700 1 --2- 172.21.15.39:0/3871526725 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f336809b680 0x7f336809db30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3374009f00 tx=0x7f3374009e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:25.689 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.687+0000 7f337effd700 1 -- 172.21.15.39:0/3871526725 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f3370090490 con 0x7f33800fccb0 2023-06-22T17:26:25.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:25 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:25.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:25 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:25.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:25 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:25.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:25 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:25.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:25 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:25.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:25 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:25.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:25 smithi039 ceph-mon[84805]: Deploying daemon mds.b on smithi101 2023-06-22T17:26:25.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:25 smithi039 ceph-mon[84805]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:25.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:25 smithi039 ceph-mon[84805]: mds.? [v2:172.21.15.39:6840/2014121640,v1:172.21.15.39:6841/2014121640] up:boot 2023-06-22T17:26:25.935 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:25 smithi039 ceph-mon[84805]: fsmap 11 up:standby 2023-06-22T17:26:25.935 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:25 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-06-22T17:26:25.937 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.936+0000 7f338826e700 1 -- 172.21.15.39:0/3871526725 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3380108600 con 0x7f33800fccb0 2023-06-22T17:26:25.938 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.936+0000 7f337effd700 1 -- 172.21.15.39:0/3871526725 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+8846 (secure 0 0 0) 0x7f3370093ab0 con 0x7f33800fccb0 2023-06-22T17:26:25.939 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:26:25.939 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3630756954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3630756954},{"type":"v1","addr":"172.21.15.39:6835","nonce":3630756954}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/768662129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":768662129},{"type":"v1","addr":"172.21.15.39:6837","nonce":768662129}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/3636987214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":3636987214},{"type":"v1","addr":"172.21.15.39:6839","nonce":3636987214}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6841/2014121640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":2014121640},{"type":"v1","addr":"172.21.15.39:6841","nonce":2014121640}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6833/3120964256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":3120964256},{"type":"v1","addr":"172.21.15.101:6833","nonce":3120964256}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/1691506923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":1691506923},{"type":"v1","addr":"172.21.15.133:6833","nonce":1691506923}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6835/4021581510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":4021581510},{"type":"v1","addr":"172.21.15.101:6835","nonce":4021581510}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6837/1721839744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6836","nonce":1721839744},{"type":"v1","addr":"172.21.15.101:6837","nonce":1721839744}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/1920373312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":1920373312},{"type":"v1","addr":"172.21.15.133:6835","nonce":1920373312}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/967175183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":967175183},{"type":"v1","addr":"172.21.15.133:6837","nonce":967175183}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24523,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6839/1006809426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1006809426},{"type":"v1","addr":"172.21.15.133:6839","nonce":1006809426}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-06-22T17:26:25.941 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.938+0000 7f338826e700 1 -- 172.21.15.39:0/3871526725 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f336809b680 msgr2=0x7f336809db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:25.941 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.938+0000 7f338826e700 1 --2- 172.21.15.39:0/3871526725 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f336809b680 0x7f336809db30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3374009f00 tx=0x7f3374009e30 comp rx=0 tx=0).stop 2023-06-22T17:26:25.941 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.938+0000 7f338826e700 1 -- 172.21.15.39:0/3871526725 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33800fccb0 msgr2=0x7f338010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:25.941 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.938+0000 7f338826e700 1 --2- 172.21.15.39:0/3871526725 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33800fccb0 0x7f338010da40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3370001e00 tx=0x7f3370009ec0 comp rx=0 tx=0).stop 2023-06-22T17:26:25.942 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.938+0000 7f338826e700 1 -- 172.21.15.39:0/3871526725 shutdown_connections 2023-06-22T17:26:25.942 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.938+0000 7f338826e700 1 --2- 172.21.15.39:0/3871526725 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f336809b680 0x7f336809db30 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:25.942 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.938+0000 7f338826e700 1 --2- 172.21.15.39:0/3871526725 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33800fe720 0x7f3380115010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:25.942 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.938+0000 7f338826e700 1 --2- 172.21.15.39:0/3871526725 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33800fccb0 0x7f338010da40 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:25.943 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.938+0000 7f338826e700 1 --2- 172.21.15.39:0/3871526725 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f33800fd600 0x7f338010df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:25.943 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.938+0000 7f338826e700 1 -- 172.21.15.39:0/3871526725 >> 172.21.15.39:0/3871526725 conn(0x7f33800f84a0 msgr2=0x7f3380101430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:25.943 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.939+0000 7f338826e700 1 -- 172.21.15.39:0/3871526725 shutdown_connections 2023-06-22T17:26:25.944 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:25.939+0000 7f338826e700 1 -- 172.21.15.39:0/3871526725 wait complete. 2023-06-22T17:26:25.944 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 12 2023-06-22T17:26:26.032 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:25 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:26.032 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:25 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:26.032 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:25 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:26.032 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:25 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:26.033 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:25 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:26.033 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:25 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:26.033 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:25 smithi101 ceph-mon[85507]: Deploying daemon mds.b on smithi101 2023-06-22T17:26:26.033 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:25 smithi101 ceph-mon[85507]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:26.033 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:25 smithi101 ceph-mon[85507]: mds.? [v2:172.21.15.39:6840/2014121640,v1:172.21.15.39:6841/2014121640] up:boot 2023-06-22T17:26:26.033 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:25 smithi101 ceph-mon[85507]: fsmap 11 up:standby 2023-06-22T17:26:26.033 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:25 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-06-22T17:26:26.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:25 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:26.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:25 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:26.172 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:25 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:26.172 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:25 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-22T17:26:26.172 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:25 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-22T17:26:26.173 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:25 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:26.173 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:25 smithi133 ceph-mon[86268]: Deploying daemon mds.b on smithi101 2023-06-22T17:26:26.173 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:25 smithi133 ceph-mon[86268]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:26.173 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:25 smithi133 ceph-mon[86268]: mds.? [v2:172.21.15.39:6840/2014121640,v1:172.21.15.39:6841/2014121640] up:boot 2023-06-22T17:26:26.174 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:25 smithi133 ceph-mon[86268]: fsmap 11 up:standby 2023-06-22T17:26:26.174 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:25 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-06-22T17:26:26.377 INFO:journalctl@ceph.mds.b.smithi101.stdout:Jun 22 17:26:26 smithi101 systemd[1]: Starting Ceph mds.b for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:26:27.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:26 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/3871526725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:27.127 INFO:journalctl@ceph.mds.b.smithi101.stdout:Jun 22 17:26:26 smithi101 ceph-mds[118194]: -- 172.21.15.101:0/1814721312 <== mon.2 v2:172.21.15.101:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556bf1af29c0 con 0x556bf1ac9000 2023-06-22T17:26:27.127 INFO:journalctl@ceph.mds.b.smithi101.stdout:Jun 22 17:26:26 smithi101 conmon[118170]: starting mds.b at 2023-06-22T17:26:27.127 INFO:journalctl@ceph.mds.b.smithi101.stdout:Jun 22 17:26:26 smithi101 bash[118129]: fb3219933de471fdc6f02c592c86f0dfa98b432bdd290d4f7622b425bdda6c46 2023-06-22T17:26:27.127 INFO:journalctl@ceph.mds.b.smithi101.stdout:Jun 22 17:26:26 smithi101 systemd[1]: Started Ceph mds.b for d88002e2-1120-11ee-9b2b-001a4aab830c. 2023-06-22T17:26:27.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:26 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/3871526725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:27.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:26 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/3871526725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:27.954 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T17:26:28.080 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.078+0000 7f5c22346700 1 -- 172.21.15.39:0/3872346005 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c1c0fd640 msgr2=0x7f5c1c103ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:28.081 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.078+0000 7f5c22346700 1 --2- 172.21.15.39:0/3872346005 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c1c0fd640 0x7f5c1c103ee0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5c14009d60 tx=0x7f5c14004d10 comp rx=0 tx=0).stop 2023-06-22T17:26:28.081 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.078+0000 7f5c22346700 1 -- 172.21.15.39:0/3872346005 shutdown_connections 2023-06-22T17:26:28.081 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.078+0000 7f5c22346700 1 --2- 172.21.15.39:0/3872346005 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c1c0fd640 0x7f5c1c103ee0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:28.081 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.078+0000 7f5c22346700 1 --2- 172.21.15.39:0/3872346005 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5c1c0fb9e0 0x7f5c1c0fbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:28.081 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.078+0000 7f5c22346700 1 --2- 172.21.15.39:0/3872346005 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5c1c0fcc90 0x7f5c1c0fd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:28.081 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.078+0000 7f5c22346700 1 -- 172.21.15.39:0/3872346005 >> 172.21.15.39:0/3872346005 conn(0x7f5c1c0f7140 msgr2=0x7f5c1c0f9570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:28.082 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.078+0000 7f5c22346700 1 -- 172.21.15.39:0/3872346005 shutdown_connections 2023-06-22T17:26:28.082 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.078+0000 7f5c22346700 1 -- 172.21.15.39:0/3872346005 wait complete. 2023-06-22T17:26:28.082 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.079+0000 7f5c22346700 1 Processor -- start 2023-06-22T17:26:28.082 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.079+0000 7f5c22346700 1 -- start start 2023-06-22T17:26:28.084 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.079+0000 7f5c22346700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5c1c0fb9e0 0x7f5c1c065a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:28.084 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.079+0000 7f5c22346700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c1c0fcc90 0x7f5c1c112370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:28.084 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.079+0000 7f5c22346700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5c1c0fd640 0x7f5c1c1149b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:28.084 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.079+0000 7f5c22346700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c115150 con 0x7f5c1c0fcc90 2023-06-22T17:26:28.085 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.079+0000 7f5c22346700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c1152c0 con 0x7f5c1c0fd640 2023-06-22T17:26:28.085 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.079+0000 7f5c22346700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c115480 con 0x7f5c1c0fb9e0 2023-06-22T17:26:28.085 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.079+0000 7f5c20b43700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c1c0fcc90 0x7f5c1c112370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:28.085 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.079+0000 7f5c21344700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5c1c0fb9e0 0x7f5c1c065a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:28.085 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.079+0000 7f5c20b43700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c1c0fcc90 0x7f5c1c112370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37244/0 (socket says 172.21.15.39:37244) 2023-06-22T17:26:28.085 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.079+0000 7f5c20b43700 1 -- 172.21.15.39:0/3584127861 learned_addr learned my addr 172.21.15.39:0/3584127861 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:26:28.085 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.079+0000 7f5c21b45700 1 --2- 172.21.15.39:0/3584127861 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5c1c0fd640 0x7f5c1c1149b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:28.086 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.079+0000 7f5c20b43700 1 -- 172.21.15.39:0/3584127861 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5c1c0fb9e0 msgr2=0x7f5c1c065a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:28.086 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.079+0000 7f5c20b43700 1 --2- 172.21.15.39:0/3584127861 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5c1c0fb9e0 0x7f5c1c065a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:28.086 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.079+0000 7f5c20b43700 1 -- 172.21.15.39:0/3584127861 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5c1c0fd640 msgr2=0x7f5c1c1149b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:28.086 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.079+0000 7f5c20b43700 1 --2- 172.21.15.39:0/3584127861 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5c1c0fd640 0x7f5c1c1149b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:28.086 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.079+0000 7f5c20b43700 1 -- 172.21.15.39:0/3584127861 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c14009940 con 0x7f5c1c0fcc90 2023-06-22T17:26:28.086 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.079+0000 7f5c21344700 1 --2- 172.21.15.39:0/3584127861 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5c1c0fb9e0 0x7f5c1c065a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T17:26:28.086 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.080+0000 7f5c20b43700 1 --2- 172.21.15.39:0/3584127861 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c1c0fcc90 0x7f5c1c112370 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5c10002ad0 tx=0x7f5c100046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:28.087 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.080+0000 7f5c19ffb700 1 -- 172.21.15.39:0/3584127861 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c10007cc0 con 0x7f5c1c0fcc90 2023-06-22T17:26:28.087 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.080+0000 7f5c19ffb700 1 -- 172.21.15.39:0/3584127861 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f5c1000f040 con 0x7f5c1c0fcc90 2023-06-22T17:26:28.087 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.080+0000 7f5c22346700 1 -- 172.21.15.39:0/3584127861 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c1c115760 con 0x7f5c1c0fcc90 2023-06-22T17:26:28.087 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.080+0000 7f5c19ffb700 1 -- 172.21.15.39:0/3584127861 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c1000d970 con 0x7f5c1c0fcc90 2023-06-22T17:26:28.087 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.080+0000 7f5c22346700 1 -- 172.21.15.39:0/3584127861 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c1c115c80 con 0x7f5c1c0fcc90 2023-06-22T17:26:28.087 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.081+0000 7f5c22346700 1 -- 172.21.15.39:0/3584127861 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c1c102630 con 0x7f5c1c0fcc90 2023-06-22T17:26:28.087 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.082+0000 7f5c19ffb700 1 -- 172.21.15.39:0/3584127861 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f5c1001c400 con 0x7f5c1c0fcc90 2023-06-22T17:26:28.088 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.082+0000 7f5c19ffb700 1 --2- 172.21.15.39:0/3584127861 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f5c0409b680 0x7f5c0409db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:28.088 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.082+0000 7f5c21344700 1 --2- 172.21.15.39:0/3584127861 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f5c0409b680 0x7f5c0409db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:28.088 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.082+0000 7f5c19ffb700 1 -- 172.21.15.39:0/3584127861 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f5c100c44e0 con 0x7f5c1c0fcc90 2023-06-22T17:26:28.088 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.083+0000 7f5c21344700 1 --2- 172.21.15.39:0/3584127861 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f5c0409b680 0x7f5c0409db30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c009fd0 tx=0x7f5c0c00b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:28.089 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.087+0000 7f5c19ffb700 1 -- 172.21.15.39:0/3584127861 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f5c10090270 con 0x7f5c1c0fcc90 2023-06-22T17:26:28.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:27 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:28.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:27 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:28.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:27 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:28.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:27 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:28.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:27 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:28.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:27 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:26:28.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:27 smithi101 ceph-mon[85507]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:28.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:27 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:28.127 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:27 smithi101 ceph-mon[85507]: mds.? [v2:172.21.15.101:6838/2392987177,v1:172.21.15.101:6839/2392987177] up:boot 2023-06-22T17:26:28.128 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:27 smithi101 ceph-mon[85507]: fsmap 12 up:standby 2023-06-22T17:26:28.128 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:27 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-06-22T17:26:28.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:27 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:28.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:27 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:28.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:27 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:28.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:27 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:28.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:27 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:28.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:27 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:26:28.172 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:27 smithi133 ceph-mon[86268]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:28.172 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:27 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:28.172 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:27 smithi133 ceph-mon[86268]: mds.? [v2:172.21.15.101:6838/2392987177,v1:172.21.15.101:6839/2392987177] up:boot 2023-06-22T17:26:28.172 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:27 smithi133 ceph-mon[86268]: fsmap 12 up:standby 2023-06-22T17:26:28.172 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:27 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-06-22T17:26:28.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:27 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:28.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:27 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:28.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:27 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:28.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:27 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:28.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:27 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:28.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:27 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:26:28.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:27 smithi039 ceph-mon[84805]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:28.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:27 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:28.184 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:27 smithi039 ceph-mon[84805]: mds.? [v2:172.21.15.101:6838/2392987177,v1:172.21.15.101:6839/2392987177] up:boot 2023-06-22T17:26:28.185 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:27 smithi039 ceph-mon[84805]: fsmap 12 up:standby 2023-06-22T17:26:28.185 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:27 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-06-22T17:26:28.333 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.332+0000 7f5c22346700 1 -- 172.21.15.39:0/3584127861 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c1c107360 con 0x7f5c1c0fcc90 2023-06-22T17:26:28.334 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.332+0000 7f5c19ffb700 1 -- 172.21.15.39:0/3584127861 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9607 (secure 0 0 0) 0x7f5c10093890 con 0x7f5c1c0fcc90 2023-06-22T17:26:28.334 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:26:28.335 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3630756954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3630756954},{"type":"v1","addr":"172.21.15.39:6835","nonce":3630756954}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/768662129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":768662129},{"type":"v1","addr":"172.21.15.39:6837","nonce":768662129}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/3636987214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":3636987214},{"type":"v1","addr":"172.21.15.39:6839","nonce":3636987214}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6841/2014121640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":2014121640},{"type":"v1","addr":"172.21.15.39:6841","nonce":2014121640}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6833/3120964256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":3120964256},{"type":"v1","addr":"172.21.15.101:6833","nonce":3120964256}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/1691506923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":1691506923},{"type":"v1","addr":"172.21.15.133:6833","nonce":1691506923}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6835/4021581510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":4021581510},{"type":"v1","addr":"172.21.15.101:6835","nonce":4021581510}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6837/1721839744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6836","nonce":1721839744},{"type":"v1","addr":"172.21.15.101:6837","nonce":1721839744}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/1920373312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":1920373312},{"type":"v1","addr":"172.21.15.133:6835","nonce":1920373312}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/967175183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":967175183},{"type":"v1","addr":"172.21.15.133:6837","nonce":967175183}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24512,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6839/2392987177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6838","nonce":2392987177},{"type":"v1","addr":"172.21.15.101:6839","nonce":2392987177}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":24523,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6839/1006809426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1006809426},{"type":"v1","addr":"172.21.15.133:6839","nonce":1006809426}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-06-22T17:26:28.337 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.334+0000 7f5c22346700 1 -- 172.21.15.39:0/3584127861 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f5c0409b680 msgr2=0x7f5c0409db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:28.337 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.334+0000 7f5c22346700 1 --2- 172.21.15.39:0/3584127861 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f5c0409b680 0x7f5c0409db30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c009fd0 tx=0x7f5c0c00b040 comp rx=0 tx=0).stop 2023-06-22T17:26:28.337 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.334+0000 7f5c22346700 1 -- 172.21.15.39:0/3584127861 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c1c0fcc90 msgr2=0x7f5c1c112370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:28.337 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.334+0000 7f5c22346700 1 --2- 172.21.15.39:0/3584127861 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c1c0fcc90 0x7f5c1c112370 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5c10002ad0 tx=0x7f5c100046b0 comp rx=0 tx=0).stop 2023-06-22T17:26:28.337 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.334+0000 7f5c22346700 1 -- 172.21.15.39:0/3584127861 shutdown_connections 2023-06-22T17:26:28.337 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.334+0000 7f5c22346700 1 --2- 172.21.15.39:0/3584127861 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f5c0409b680 0x7f5c0409db30 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:28.337 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.334+0000 7f5c22346700 1 --2- 172.21.15.39:0/3584127861 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5c1c0fd640 0x7f5c1c1149b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:28.338 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.334+0000 7f5c22346700 1 --2- 172.21.15.39:0/3584127861 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5c1c0fb9e0 0x7f5c1c065a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:28.338 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.334+0000 7f5c22346700 1 --2- 172.21.15.39:0/3584127861 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c1c0fcc90 0x7f5c1c112370 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:28.338 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.334+0000 7f5c22346700 1 -- 172.21.15.39:0/3584127861 >> 172.21.15.39:0/3584127861 conn(0x7f5c1c0f7140 msgr2=0x7f5c1c0f8f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:28.338 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.334+0000 7f5c22346700 1 -- 172.21.15.39:0/3584127861 shutdown_connections 2023-06-22T17:26:28.338 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.334+0000 7f5c22346700 1 -- 172.21.15.39:0/3584127861 wait complete. 2023-06-22T17:26:28.339 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 13 2023-06-22T17:26:28.349 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-22T17:26:28.471 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.469+0000 7fa06df65700 1 -- 172.21.15.39:0/2005600930 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0680fe720 msgr2=0x7fa068105240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:28.471 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.469+0000 7fa06df65700 1 --2- 172.21.15.39:0/2005600930 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0680fe720 0x7fa068105240 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa0600049c0 tx=0x7fa060009e30 comp rx=0 tx=0).stop 2023-06-22T17:26:28.472 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.469+0000 7fa06df65700 1 -- 172.21.15.39:0/2005600930 shutdown_connections 2023-06-22T17:26:28.472 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.469+0000 7fa06df65700 1 --2- 172.21.15.39:0/2005600930 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0680fe720 0x7fa068105240 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:28.472 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.469+0000 7fa06df65700 1 --2- 172.21.15.39:0/2005600930 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0680fccb0 0x7fa0680fd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:28.472 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.469+0000 7fa06df65700 1 --2- 172.21.15.39:0/2005600930 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa0680fd600 0x7fa0680fdff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:28.472 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.469+0000 7fa06df65700 1 -- 172.21.15.39:0/2005600930 >> 172.21.15.39:0/2005600930 conn(0x7fa0680f84a0 msgr2=0x7fa0680fa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:28.472 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.469+0000 7fa06df65700 1 -- 172.21.15.39:0/2005600930 shutdown_connections 2023-06-22T17:26:28.472 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.469+0000 7fa06df65700 1 -- 172.21.15.39:0/2005600930 wait complete. 2023-06-22T17:26:28.473 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.469+0000 7fa06df65700 1 Processor -- start 2023-06-22T17:26:28.474 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.470+0000 7fa06df65700 1 -- start start 2023-06-22T17:26:28.475 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.470+0000 7fa06df65700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa0680fccb0 0x7fa06810d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:28.475 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.470+0000 7fa06df65700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0680fd600 0x7fa06810dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:28.475 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.470+0000 7fa06df65700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0680fe720 0x7fa068114d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:28.475 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.470+0000 7fa06df65700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa068115350 con 0x7fa0680fe720 2023-06-22T17:26:28.475 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.470+0000 7fa06df65700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0681154c0 con 0x7fa0680fccb0 2023-06-22T17:26:28.475 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.470+0000 7fa06df65700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa068115680 con 0x7fa0680fd600 2023-06-22T17:26:28.476 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.470+0000 7fa067fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0680fe720 0x7fa068114d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:28.476 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.470+0000 7fa067fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0680fe720 0x7fa068114d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37260/0 (socket says 172.21.15.39:37260) 2023-06-22T17:26:28.476 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.470+0000 7fa067fff700 1 -- 172.21.15.39:0/3426631730 learned_addr learned my addr 172.21.15.39:0/3426631730 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:26:28.476 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.470+0000 7fa066ffd700 1 --2- 172.21.15.39:0/3426631730 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0680fd600 0x7fa06810dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:28.476 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.470+0000 7fa0677fe700 1 --2- 172.21.15.39:0/3426631730 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa0680fccb0 0x7fa06810d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:28.476 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.470+0000 7fa067fff700 1 -- 172.21.15.39:0/3426631730 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0680fd600 msgr2=0x7fa06810dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:28.477 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.470+0000 7fa067fff700 1 --2- 172.21.15.39:0/3426631730 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0680fd600 0x7fa06810dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:28.477 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.470+0000 7fa067fff700 1 -- 172.21.15.39:0/3426631730 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa0680fccb0 msgr2=0x7fa06810d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:28.477 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.470+0000 7fa067fff700 1 --2- 172.21.15.39:0/3426631730 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa0680fccb0 0x7fa06810d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:28.477 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.470+0000 7fa067fff700 1 -- 172.21.15.39:0/3426631730 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa060005040 con 0x7fa0680fe720 2023-06-22T17:26:28.477 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.470+0000 7fa067fff700 1 --2- 172.21.15.39:0/3426631730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0680fe720 0x7fa068114d40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa060005010 tx=0x7fa060009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:28.477 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.470+0000 7fa05ffff700 1 -- 172.21.15.39:0/3426631730 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa06000e030 con 0x7fa0680fe720 2023-06-22T17:26:28.477 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.470+0000 7fa06df65700 1 -- 172.21.15.39:0/3426631730 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa068115900 con 0x7fa0680fe720 2023-06-22T17:26:28.477 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.470+0000 7fa05ffff700 1 -- 172.21.15.39:0/3426631730 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7fa060004160 con 0x7fa0680fe720 2023-06-22T17:26:28.478 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.470+0000 7fa05ffff700 1 -- 172.21.15.39:0/3426631730 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa060012770 con 0x7fa0680fe720 2023-06-22T17:26:28.478 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.471+0000 7fa06df65700 1 -- 172.21.15.39:0/3426631730 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa068115d60 con 0x7fa0680fe720 2023-06-22T17:26:28.478 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.471+0000 7fa06df65700 1 -- 172.21.15.39:0/3426631730 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa068103990 con 0x7fa0680fe720 2023-06-22T17:26:28.478 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.472+0000 7fa05ffff700 1 -- 172.21.15.39:0/3426631730 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7fa060012ba0 con 0x7fa0680fe720 2023-06-22T17:26:28.478 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.473+0000 7fa05ffff700 1 --2- 172.21.15.39:0/3426631730 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fa05409b680 0x7fa05409db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:28.479 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.473+0000 7fa0677fe700 1 --2- 172.21.15.39:0/3426631730 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fa05409b680 0x7fa05409db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:28.479 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.473+0000 7fa05ffff700 1 -- 172.21.15.39:0/3426631730 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fa0600c6ab0 con 0x7fa0680fe720 2023-06-22T17:26:28.479 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.473+0000 7fa0677fe700 1 --2- 172.21.15.39:0/3426631730 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fa05409b680 0x7fa05409db30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa058004b30 tx=0x7fa05800b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:28.479 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.476+0000 7fa05ffff700 1 -- 172.21.15.39:0/3426631730 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fa060092790 con 0x7fa0680fe720 2023-06-22T17:26:28.676 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.674+0000 7fa06df65700 1 -- 172.21.15.39:0/3426631730 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa068108630 con 0x7fa0680fe720 2023-06-22T17:26:28.677 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.675+0000 7fa05ffff700 1 -- 172.21.15.39:0/3426631730 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+18158 (secure 0 0 0) 0x7fa060004490 con 0x7fa0680fe720 2023-06-22T17:26:28.678 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:26:28.678 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":77,"fsid":"d88002e2-1120-11ee-9b2b-001a4aab830c","created":"2023-06-22T17:19:36.313754+0000","modified":"2023-06-22T17:25:52.036913+0000","last_up_change":"2023-06-22T17:25:51.023671+0000","last_in_change":"2023-06-22T17:25:33.400238+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-22T17:22:35.797853+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"eb8f5cba-9266-47ec-8af1-8e9edd6ec96a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":453150307},{"type":"v1","addr":"172.21.15.39:6803","nonce":453150307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":453150307},{"type":"v1","addr":"172.21.15.39:6805","nonce":453150307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":453150307},{"type":"v1","addr":"172.21.15.39:6809","nonce":453150307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":453150307},{"type":"v1","addr":"172.21.15.39:6807","nonce":453150307}]},"public_addr":"172.21.15.39:6803/453150307","cluster_addr":"172.21.15.39:6805/453150307","heartbeat_back_addr":"172.21.15.39:6809/453150307","heartbeat_front_addr":"172.21.15.39:6807/453150307","state":["exists","up"]},{"osd":1,"uuid":"8b8dbe3e-fd72-4f92-98a2-e212df5366c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6800","nonce":2708845767},{"type":"v1","addr":"172.21.15.101:6801","nonce":2708845767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2708845767},{"type":"v1","addr":"172.21.15.101:6803","nonce":2708845767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2708845767},{"type":"v1","addr":"172.21.15.101:6807","nonce":2708845767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2708845767},{"type":"v1","addr":"172.21.15.101:6805","nonce":2708845767}]},"public_addr":"172.21.15.101:6801/2708845767","cluster_addr":"172.21.15.101:6803/2708845767","heartbeat_back_addr":"172.21.15.101:6807/2708845767","heartbeat_front_addr":"172.21.15.101:6805/2708845767","state":["exists","up"]},{"osd":2,"uuid":"7b25ade6-b5e2-4194-92c7-3baf8c4e9560","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":3632902853},{"type":"v1","addr":"172.21.15.133:6801","nonce":3632902853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":3632902853},{"type":"v1","addr":"172.21.15.133:6803","nonce":3632902853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":3632902853},{"type":"v1","addr":"172.21.15.133:6807","nonce":3632902853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":3632902853},{"type":"v1","addr":"172.21.15.133:6805","nonce":3632902853}]},"public_addr":"172.21.15.133:6801/3632902853","cluster_addr":"172.21.15.133:6803/3632902853","heartbeat_back_addr":"172.21.15.133:6807/3632902853","heartbeat_front_addr":"172.21.15.133:6805/3632902853","state":["exists","up"]},{"osd":3,"uuid":"b91b9111-cb8f-4c51-85c6-96c0851d57d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1215819990},{"type":"v1","addr":"172.21.15.39:6811","nonce":1215819990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1215819990},{"type":"v1","addr":"172.21.15.39:6813","nonce":1215819990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":1215819990},{"type":"v1","addr":"172.21.15.39:6817","nonce":1215819990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1215819990},{"type":"v1","addr":"172.21.15.39:6815","nonce":1215819990}]},"public_addr":"172.21.15.39:6811/1215819990","cluster_addr":"172.21.15.39:6813/1215819990","heartbeat_back_addr":"172.21.15.39:6817/1215819990","heartbeat_front_addr":"172.21.15.39:6815/1215819990","state":["exists","up"]},{"osd":4,"uuid":"14f6d038-1e3c-41e0-9be6-3615099ac5fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":1371317064},{"type":"v1","addr":"172.21.15.101:6809","nonce":1371317064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":1371317064},{"type":"v1","addr":"172.21.15.101:6811","nonce":1371317064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":1371317064},{"type":"v1","addr":"172.21.15.101:6815","nonce":1371317064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":1371317064},{"type":"v1","addr":"172.21.15.101:6813","nonce":1371317064}]},"public_addr":"172.21.15.101:6809/1371317064","cluster_addr":"172.21.15.101:6811/1371317064","heartbeat_back_addr":"172.21.15.101:6815/1371317064","heartbeat_front_addr":"172.21.15.101:6813/1371317064","state":["exists","up"]},{"osd":5,"uuid":"6daaeb92-c73d-4c30-bb1d-18d5305a63a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":747181981},{"type":"v1","addr":"172.21.15.133:6809","nonce":747181981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":747181981},{"type":"v1","addr":"172.21.15.133:6811","nonce":747181981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":747181981},{"type":"v1","addr":"172.21.15.133:6815","nonce":747181981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":747181981},{"type":"v1","addr":"172.21.15.133:6813","nonce":747181981}]},"public_addr":"172.21.15.133:6809/747181981","cluster_addr":"172.21.15.133:6811/747181981","heartbeat_back_addr":"172.21.15.133:6815/747181981","heartbeat_front_addr":"172.21.15.133:6813/747181981","state":["exists","up"]},{"osd":6,"uuid":"21f46569-8123-4bdd-b3c5-4cc8bc271899","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1221216882},{"type":"v1","addr":"172.21.15.39:6819","nonce":1221216882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1221216882},{"type":"v1","addr":"172.21.15.39:6821","nonce":1221216882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1221216882},{"type":"v1","addr":"172.21.15.39:6825","nonce":1221216882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1221216882},{"type":"v1","addr":"172.21.15.39:6823","nonce":1221216882}]},"public_addr":"172.21.15.39:6819/1221216882","cluster_addr":"172.21.15.39:6821/1221216882","heartbeat_back_addr":"172.21.15.39:6825/1221216882","heartbeat_front_addr":"172.21.15.39:6823/1221216882","state":["exists","up"]},{"osd":7,"uuid":"7a1cbc1b-8cf0-4153-b8e2-3d3313bed2ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":1606613660},{"type":"v1","addr":"172.21.15.101:6817","nonce":1606613660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":1606613660},{"type":"v1","addr":"172.21.15.101:6819","nonce":1606613660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":1606613660},{"type":"v1","addr":"172.21.15.101:6823","nonce":1606613660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":1606613660},{"type":"v1","addr":"172.21.15.101:6821","nonce":1606613660}]},"public_addr":"172.21.15.101:6817/1606613660","cluster_addr":"172.21.15.101:6819/1606613660","heartbeat_back_addr":"172.21.15.101:6823/1606613660","heartbeat_front_addr":"172.21.15.101:6821/1606613660","state":["exists","up"]},{"osd":8,"uuid":"ab09136e-4326-4c02-ac59-3d2264e85b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":1363381164},{"type":"v1","addr":"172.21.15.133:6817","nonce":1363381164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":1363381164},{"type":"v1","addr":"172.21.15.133:6819","nonce":1363381164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":1363381164},{"type":"v1","addr":"172.21.15.133:6823","nonce":1363381164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":1363381164},{"type":"v1","addr":"172.21.15.133:6821","nonce":1363381164}]},"public_addr":"172.21.15.133:6817/1363381164","cluster_addr":"172.21.15.133:6819/1363381164","heartbeat_back_addr":"172.21.15.133:6823/1363381164","heartbeat_front_addr":"172.21.15.133:6821/1363381164","state":["exists","up"]},{"osd":9,"uuid":"ec8c1344-0478-4094-b544-3360054fd92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3010986736},{"type":"v1","addr":"172.21.15.39:6827","nonce":3010986736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3010986736},{"type":"v1","addr":"172.21.15.39:6829","nonce":3010986736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3010986736},{"type":"v1","addr":"172.21.15.39:6833","nonce":3010986736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3010986736},{"type":"v1","addr":"172.21.15.39:6831","nonce":3010986736}]},"public_addr":"172.21.15.39:6827/3010986736","cluster_addr":"172.21.15.39:6829/3010986736","heartbeat_back_addr":"172.21.15.39:6833/3010986736","heartbeat_front_addr":"172.21.15.39:6831/3010986736","state":["exists","up"]},{"osd":10,"uuid":"9da2943f-b947-49d6-bab1-bc3a95299996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3982067769},{"type":"v1","addr":"172.21.15.101:6825","nonce":3982067769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":3982067769},{"type":"v1","addr":"172.21.15.101:6827","nonce":3982067769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":3982067769},{"type":"v1","addr":"172.21.15.101:6831","nonce":3982067769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":3982067769},{"type":"v1","addr":"172.21.15.101:6829","nonce":3982067769}]},"public_addr":"172.21.15.101:6825/3982067769","cluster_addr":"172.21.15.101:6827/3982067769","heartbeat_back_addr":"172.21.15.101:6831/3982067769","heartbeat_front_addr":"172.21.15.101:6829/3982067769","state":["exists","up"]},{"osd":11,"uuid":"ee5d51b4-c9ad-4603-a967-c60f3c1f2cdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":1265049047},{"type":"v1","addr":"172.21.15.133:6825","nonce":1265049047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6826","nonce":1265049047},{"type":"v1","addr":"172.21.15.133:6827","nonce":1265049047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6830","nonce":1265049047},{"type":"v1","addr":"172.21.15.133:6831","nonce":1265049047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6828","nonce":1265049047},{"type":"v1","addr":"172.21.15.133:6829","nonce":1265049047}]},"public_addr":"172.21.15.133:6825/1265049047","cluster_addr":"172.21.15.133:6827/1265049047","heartbeat_back_addr":"172.21.15.133:6831/1265049047","heartbeat_front_addr":"172.21.15.133:6829/1265049047","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:21:49.604087+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:22:11.298409+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:22:32.913915+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:22:56.095316+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:23:17.505593+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:23:38.490450+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:24:01.498630+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:24:22.952502+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:24:44.480122+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:25:07.141899+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:25:28.109970+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:25:49.227776+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:6801/1350590065":"2023-06-23T17:20:27.281966+0000","172.21.15.39:6800/1350590065":"2023-06-23T17:20:27.281966+0000","172.21.15.39:0/82302526":"2023-06-23T17:20:27.281966+0000","172.21.15.39:0/260892351":"2023-06-23T17:20:05.206927+0000","172.21.15.39:0/1513096987":"2023-06-23T17:20:05.206927+0000","172.21.15.39:6800/3960151496":"2023-06-23T17:20:05.206927+0000","172.21.15.39:6801/3960151496":"2023-06-23T17:20:05.206927+0000","172.21.15.39:0/2480502158":"2023-06-23T17:20:27.281966+0000","172.21.15.39:0/3848722689":"2023-06-23T17:20:27.281966+0000","172.21.15.39:0/2257210574":"2023-06-23T17:20:05.206927+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T17:26:28.680 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.677+0000 7fa06df65700 1 -- 172.21.15.39:0/3426631730 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fa05409b680 msgr2=0x7fa05409db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:28.680 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.677+0000 7fa06df65700 1 --2- 172.21.15.39:0/3426631730 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fa05409b680 0x7fa05409db30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa058004b30 tx=0x7fa05800b040 comp rx=0 tx=0).stop 2023-06-22T17:26:28.680 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.678+0000 7fa06df65700 1 -- 172.21.15.39:0/3426631730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0680fe720 msgr2=0x7fa068114d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:28.680 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.678+0000 7fa06df65700 1 --2- 172.21.15.39:0/3426631730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0680fe720 0x7fa068114d40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa060005010 tx=0x7fa060009e30 comp rx=0 tx=0).stop 2023-06-22T17:26:28.680 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.678+0000 7fa06df65700 1 -- 172.21.15.39:0/3426631730 shutdown_connections 2023-06-22T17:26:28.681 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.678+0000 7fa06df65700 1 --2- 172.21.15.39:0/3426631730 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7fa05409b680 0x7fa05409db30 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:28.681 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.678+0000 7fa06df65700 1 --2- 172.21.15.39:0/3426631730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0680fe720 0x7fa068114d40 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:28.681 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.678+0000 7fa06df65700 1 --2- 172.21.15.39:0/3426631730 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa0680fccb0 0x7fa06810d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:28.681 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.678+0000 7fa06df65700 1 --2- 172.21.15.39:0/3426631730 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0680fd600 0x7fa06810dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:28.681 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.678+0000 7fa06df65700 1 -- 172.21.15.39:0/3426631730 >> 172.21.15.39:0/3426631730 conn(0x7fa0680f84a0 msgr2=0x7fa0680fa300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:28.681 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.678+0000 7fa06df65700 1 -- 172.21.15.39:0/3426631730 shutdown_connections 2023-06-22T17:26:28.681 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.678+0000 7fa06df65700 1 -- 172.21.15.39:0/3426631730 wait complete. 2023-06-22T17:26:28.693 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-22T17:22:35.797853+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-06-22T17:26:28.694 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-22T17:26:28.822 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.820+0000 7f8d99c8c700 1 -- 172.21.15.39:0/777377223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d94102aa0 msgr2=0x7f8d94104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:28.823 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.820+0000 7f8d99c8c700 1 --2- 172.21.15.39:0/777377223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d94102aa0 0x7f8d94104e80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c009d60 tx=0x7f8d8c004d10 comp rx=0 tx=0).stop 2023-06-22T17:26:28.823 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.820+0000 7f8d99c8c700 1 -- 172.21.15.39:0/777377223 shutdown_connections 2023-06-22T17:26:28.823 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.820+0000 7f8d99c8c700 1 --2- 172.21.15.39:0/777377223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d94102aa0 0x7f8d94104e80 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:28.823 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.820+0000 7f8d99c8c700 1 --2- 172.21.15.39:0/777377223 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8d940fd860 0x7f8d940ffc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:28.823 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.820+0000 7f8d99c8c700 1 --2- 172.21.15.39:0/777377223 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8d94100180 0x7f8d94102560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:28.824 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.820+0000 7f8d99c8c700 1 -- 172.21.15.39:0/777377223 >> 172.21.15.39:0/777377223 conn(0x7f8d940f7380 msgr2=0x7f8d940f97b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:28.824 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.820+0000 7f8d99c8c700 1 -- 172.21.15.39:0/777377223 shutdown_connections 2023-06-22T17:26:28.824 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.820+0000 7f8d99c8c700 1 -- 172.21.15.39:0/777377223 wait complete. 2023-06-22T17:26:28.824 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.821+0000 7f8d99c8c700 1 Processor -- start 2023-06-22T17:26:28.826 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.821+0000 7f8d99c8c700 1 -- start start 2023-06-22T17:26:28.826 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.821+0000 7f8d99c8c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8d940fd860 0x7f8d9410d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:28.826 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.821+0000 7f8d99c8c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8d94100180 0x7f8d9410dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:28.826 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.821+0000 7f8d99c8c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d94102aa0 0x7f8d94114d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:28.826 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.821+0000 7f8d99c8c700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d94115310 con 0x7f8d94102aa0 2023-06-22T17:26:28.827 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.821+0000 7f8d99c8c700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d94115480 con 0x7f8d94100180 2023-06-22T17:26:28.827 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.821+0000 7f8d99c8c700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d94115640 con 0x7f8d940fd860 2023-06-22T17:26:28.827 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.821+0000 7f8d93fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d94102aa0 0x7f8d94114d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:28.827 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.821+0000 7f8d937fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8d940fd860 0x7f8d9410d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:28.827 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.821+0000 7f8d93fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d94102aa0 0x7f8d94114d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37284/0 (socket says 172.21.15.39:37284) 2023-06-22T17:26:28.827 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.821+0000 7f8d93fff700 1 -- 172.21.15.39:0/4277973365 learned_addr learned my addr 172.21.15.39:0/4277973365 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:26:28.827 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.821+0000 7f8d92ffd700 1 --2- 172.21.15.39:0/4277973365 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8d94100180 0x7f8d9410dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:28.828 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.822+0000 7f8d93fff700 1 -- 172.21.15.39:0/4277973365 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8d940fd860 msgr2=0x7f8d9410d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:28.828 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.822+0000 7f8d93fff700 1 --2- 172.21.15.39:0/4277973365 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8d940fd860 0x7f8d9410d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:28.828 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.822+0000 7f8d93fff700 1 -- 172.21.15.39:0/4277973365 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8d94100180 msgr2=0x7f8d9410dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:28.828 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.822+0000 7f8d93fff700 1 --2- 172.21.15.39:0/4277973365 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8d94100180 0x7f8d9410dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:28.828 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.822+0000 7f8d93fff700 1 -- 172.21.15.39:0/4277973365 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d8c009940 con 0x7f8d94102aa0 2023-06-22T17:26:28.828 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.822+0000 7f8d937fe700 1 --2- 172.21.15.39:0/4277973365 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8d940fd860 0x7f8d9410d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T17:26:28.829 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.822+0000 7f8d93fff700 1 --2- 172.21.15.39:0/4277973365 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d94102aa0 0x7f8d94114d00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c0096a0 tx=0x7f8d8c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:28.829 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.822+0000 7f8d7bfff700 1 -- 172.21.15.39:0/4277973365 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c00e040 con 0x7f8d94102aa0 2023-06-22T17:26:28.829 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.822+0000 7f8d99c8c700 1 -- 172.21.15.39:0/4277973365 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d941158c0 con 0x7f8d94102aa0 2023-06-22T17:26:28.829 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.822+0000 7f8d7bfff700 1 -- 172.21.15.39:0/4277973365 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f8d8c007c20 con 0x7f8d94102aa0 2023-06-22T17:26:28.829 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.822+0000 7f8d7bfff700 1 -- 172.21.15.39:0/4277973365 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c01c7c0 con 0x7f8d94102aa0 2023-06-22T17:26:28.829 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.822+0000 7f8d99c8c700 1 -- 172.21.15.39:0/4277973365 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d94115df0 con 0x7f8d94102aa0 2023-06-22T17:26:28.829 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.823+0000 7f8d99c8c700 1 -- 172.21.15.39:0/4277973365 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d940f8eb0 con 0x7f8d94102aa0 2023-06-22T17:26:28.829 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.824+0000 7f8d7bfff700 1 -- 172.21.15.39:0/4277973365 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f8d8c019070 con 0x7f8d94102aa0 2023-06-22T17:26:28.830 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.825+0000 7f8d7bfff700 1 --2- 172.21.15.39:0/4277973365 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f8d7c09b730 0x7f8d7c09dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:28.831 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.825+0000 7f8d937fe700 1 --2- 172.21.15.39:0/4277973365 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f8d7c09b730 0x7f8d7c09dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:28.831 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.825+0000 7f8d7bfff700 1 -- 172.21.15.39:0/4277973365 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f8d8c0c69a0 con 0x7f8d94102aa0 2023-06-22T17:26:28.831 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.825+0000 7f8d937fe700 1 --2- 172.21.15.39:0/4277973365 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f8d7c09b730 0x7f8d7c09dbe0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8d84010520 tx=0x7f8d8400b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:28.832 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:28.829+0000 7f8d7bfff700 1 -- 172.21.15.39:0/4277973365 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f8d8c0926a0 con 0x7f8d94102aa0 2023-06-22T17:26:29.046 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.044+0000 7f8d99c8c700 1 -- 172.21.15.39:0/4277973365 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8d9402aff0 con 0x7f8d94102aa0 2023-06-22T17:26:29.047 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.044+0000 7f8d7bfff700 1 -- 172.21.15.39:0/4277973365 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8d8c095cc0 con 0x7f8d94102aa0 2023-06-22T17:26:29.047 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2023-06-22T17:26:29.049 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.046+0000 7f8d99c8c700 1 -- 172.21.15.39:0/4277973365 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f8d7c09b730 msgr2=0x7f8d7c09dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:29.049 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.046+0000 7f8d99c8c700 1 --2- 172.21.15.39:0/4277973365 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f8d7c09b730 0x7f8d7c09dbe0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8d84010520 tx=0x7f8d8400b040 comp rx=0 tx=0).stop 2023-06-22T17:26:29.049 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.046+0000 7f8d99c8c700 1 -- 172.21.15.39:0/4277973365 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d94102aa0 msgr2=0x7f8d94114d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:29.050 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.046+0000 7f8d99c8c700 1 --2- 172.21.15.39:0/4277973365 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d94102aa0 0x7f8d94114d00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c0096a0 tx=0x7f8d8c004d10 comp rx=0 tx=0).stop 2023-06-22T17:26:29.050 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.046+0000 7f8d99c8c700 1 -- 172.21.15.39:0/4277973365 shutdown_connections 2023-06-22T17:26:29.050 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.046+0000 7f8d99c8c700 1 --2- 172.21.15.39:0/4277973365 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f8d7c09b730 0x7f8d7c09dbe0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:29.050 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.046+0000 7f8d99c8c700 1 --2- 172.21.15.39:0/4277973365 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d94102aa0 0x7f8d94114d00 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:29.050 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.046+0000 7f8d99c8c700 1 --2- 172.21.15.39:0/4277973365 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8d940fd860 0x7f8d9410d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:29.051 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.046+0000 7f8d99c8c700 1 --2- 172.21.15.39:0/4277973365 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8d94100180 0x7f8d9410dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:29.051 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.046+0000 7f8d99c8c700 1 -- 172.21.15.39:0/4277973365 >> 172.21.15.39:0/4277973365 conn(0x7f8d940f7380 msgr2=0x7f8d940fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:29.051 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.046+0000 7f8d99c8c700 1 -- 172.21.15.39:0/4277973365 shutdown_connections 2023-06-22T17:26:29.051 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.046+0000 7f8d99c8c700 1 -- 172.21.15.39:0/4277973365 wait complete. 2023-06-22T17:26:29.059 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:28 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/3584127861' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:29.060 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:28 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/3426631730' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-06-22T17:26:29.061 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-06-22T17:26:29.170 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:28 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/3584127861' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:29.171 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:28 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/3426631730' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-06-22T17:26:29.186 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.183+0000 7f832c2cb700 1 -- 172.21.15.39:0/582642849 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83240fce20 msgr2=0x7f83240fd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:29.186 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.183+0000 7f832c2cb700 1 --2- 172.21.15.39:0/582642849 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83240fce20 0x7f83240fd270 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f83140049c0 tx=0x7f8314009e30 comp rx=0 tx=0).stop 2023-06-22T17:26:29.186 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.184+0000 7f832c2cb700 1 -- 172.21.15.39:0/582642849 shutdown_connections 2023-06-22T17:26:29.186 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.184+0000 7f832c2cb700 1 --2- 172.21.15.39:0/582642849 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83240fd7b0 0x7f8324104070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:29.186 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.184+0000 7f832c2cb700 1 --2- 172.21.15.39:0/582642849 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f83240fbc10 0x7f83240fc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:29.187 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.184+0000 7f832c2cb700 1 --2- 172.21.15.39:0/582642849 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83240fce20 0x7f83240fd270 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:29.187 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.184+0000 7f832c2cb700 1 -- 172.21.15.39:0/582642849 >> 172.21.15.39:0/582642849 conn(0x7f83240f7350 msgr2=0x7f83240f97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:29.187 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.184+0000 7f832c2cb700 1 -- 172.21.15.39:0/582642849 shutdown_connections 2023-06-22T17:26:29.187 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.184+0000 7f832c2cb700 1 -- 172.21.15.39:0/582642849 wait complete. 2023-06-22T17:26:29.189 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.184+0000 7f832c2cb700 1 Processor -- start 2023-06-22T17:26:29.189 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.184+0000 7f832c2cb700 1 -- start start 2023-06-22T17:26:29.189 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.185+0000 7f832c2cb700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83240fbc10 0x7f832410c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:29.189 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.185+0000 7f832c2cb700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f83240fce20 0x7f832410cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:29.190 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.185+0000 7f832c2cb700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83240fd7b0 0x7f8324113e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:29.190 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.185+0000 7f832c2cb700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8324114480 con 0x7f83240fbc10 2023-06-22T17:26:29.190 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.185+0000 7f832c2cb700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83241145f0 con 0x7f83240fce20 2023-06-22T17:26:29.190 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.185+0000 7f832c2cb700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83241147b0 con 0x7f83240fd7b0 2023-06-22T17:26:29.190 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.185+0000 7f832a067700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83240fbc10 0x7f832410c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:29.190 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.185+0000 7f832a067700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83240fbc10 0x7f832410c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37314/0 (socket says 172.21.15.39:37314) 2023-06-22T17:26:29.190 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.185+0000 7f832a067700 1 -- 172.21.15.39:0/1290518146 learned_addr learned my addr 172.21.15.39:0/1290518146 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:26:29.190 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.185+0000 7f832a868700 1 --2- 172.21.15.39:0/1290518146 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83240fd7b0 0x7f8324113e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:29.191 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.185+0000 7f8329866700 1 --2- 172.21.15.39:0/1290518146 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f83240fce20 0x7f832410cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:29.191 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.185+0000 7f832a067700 1 -- 172.21.15.39:0/1290518146 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83240fd7b0 msgr2=0x7f8324113e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:29.191 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.185+0000 7f832a067700 1 --2- 172.21.15.39:0/1290518146 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83240fd7b0 0x7f8324113e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:29.191 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.185+0000 7f832a067700 1 -- 172.21.15.39:0/1290518146 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f83240fce20 msgr2=0x7f832410cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:29.191 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.185+0000 7f832a067700 1 --2- 172.21.15.39:0/1290518146 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f83240fce20 0x7f832410cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:29.191 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.185+0000 7f832a067700 1 -- 172.21.15.39:0/1290518146 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8314005040 con 0x7f83240fbc10 2023-06-22T17:26:29.191 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.185+0000 7f8329866700 1 --2- 172.21.15.39:0/1290518146 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f83240fce20 0x7f832410cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-22T17:26:29.191 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.185+0000 7f832a067700 1 --2- 172.21.15.39:0/1290518146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83240fbc10 0x7f832410c8a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8320010e00 tx=0x7f832000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:29.192 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.185+0000 7f831affd700 1 -- 172.21.15.39:0/1290518146 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83200047d0 con 0x7f83240fbc10 2023-06-22T17:26:29.192 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.185+0000 7f832c2cb700 1 -- 172.21.15.39:0/1290518146 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8324065c10 con 0x7f83240fbc10 2023-06-22T17:26:29.192 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.185+0000 7f831affd700 1 -- 172.21.15.39:0/1290518146 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f832000ed60 con 0x7f83240fbc10 2023-06-22T17:26:29.192 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.186+0000 7f831affd700 1 -- 172.21.15.39:0/1290518146 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8320012430 con 0x7f83240fbc10 2023-06-22T17:26:29.192 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.186+0000 7f832c2cb700 1 -- 172.21.15.39:0/1290518146 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8324066150 con 0x7f83240fbc10 2023-06-22T17:26:29.192 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.187+0000 7f832c2cb700 1 -- 172.21.15.39:0/1290518146 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8324107540 con 0x7f83240fbc10 2023-06-22T17:26:29.193 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.188+0000 7f831affd700 1 -- 172.21.15.39:0/1290518146 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f8320018030 con 0x7f83240fbc10 2023-06-22T17:26:29.193 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.188+0000 7f831affd700 1 --2- 172.21.15.39:0/1290518146 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f831009b680 0x7f831009db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:29.193 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.188+0000 7f8329866700 1 --2- 172.21.15.39:0/1290518146 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f831009b680 0x7f831009db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:29.194 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.188+0000 7f831affd700 1 -- 172.21.15.39:0/1290518146 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f8320016070 con 0x7f83240fbc10 2023-06-22T17:26:29.194 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.189+0000 7f8329866700 1 --2- 172.21.15.39:0/1290518146 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f831009b680 0x7f831009db30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8314007f50 tx=0x7f8314009e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:29.194 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.192+0000 7f831affd700 1 -- 172.21.15.39:0/1290518146 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f8320092970 con 0x7f83240fbc10 2023-06-22T17:26:29.376 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:28 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/3584127861' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-22T17:26:29.377 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:28 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/3426631730' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-06-22T17:26:29.396 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.394+0000 7f832c2cb700 1 -- 172.21.15.39:0/1290518146 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f83241076d0 con 0x7f83240fbc10 2023-06-22T17:26:29.396 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.394+0000 7f831affd700 1 -- 172.21.15.39:0/1290518146 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v77) v1 ==== 84+0+65 (secure 0 0 0) 0x7f83241076d0 con 0x7f83240fbc10 2023-06-22T17:26:29.396 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:26:29.397 INFO:teuthology.orchestra.run.smithi039.stdout:[ 2023-06-22T17:26:29.397 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-06-22T17:26:29.397 INFO:teuthology.orchestra.run.smithi039.stdout: "poolnum": 1, 2023-06-22T17:26:29.397 INFO:teuthology.orchestra.run.smithi039.stdout: "poolname": ".mgr" 2023-06-22T17:26:29.397 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-06-22T17:26:29.397 INFO:teuthology.orchestra.run.smithi039.stdout:] 2023-06-22T17:26:29.398 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.396+0000 7f832c2cb700 1 -- 172.21.15.39:0/1290518146 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f831009b680 msgr2=0x7f831009db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:29.399 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.396+0000 7f832c2cb700 1 --2- 172.21.15.39:0/1290518146 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f831009b680 0x7f831009db30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8314007f50 tx=0x7f8314009e30 comp rx=0 tx=0).stop 2023-06-22T17:26:29.399 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.396+0000 7f832c2cb700 1 -- 172.21.15.39:0/1290518146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83240fbc10 msgr2=0x7f832410c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:29.399 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.396+0000 7f832c2cb700 1 --2- 172.21.15.39:0/1290518146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83240fbc10 0x7f832410c8a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8320010e00 tx=0x7f832000b040 comp rx=0 tx=0).stop 2023-06-22T17:26:29.399 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.396+0000 7f832c2cb700 1 -- 172.21.15.39:0/1290518146 shutdown_connections 2023-06-22T17:26:29.399 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.396+0000 7f832c2cb700 1 --2- 172.21.15.39:0/1290518146 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f831009b680 0x7f831009db30 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:29.399 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.396+0000 7f832c2cb700 1 --2- 172.21.15.39:0/1290518146 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83240fd7b0 0x7f8324113e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:29.400 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.396+0000 7f832c2cb700 1 --2- 172.21.15.39:0/1290518146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83240fbc10 0x7f832410c8a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:29.400 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.396+0000 7f832c2cb700 1 --2- 172.21.15.39:0/1290518146 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f83240fce20 0x7f832410cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:29.400 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.397+0000 7f832c2cb700 1 -- 172.21.15.39:0/1290518146 >> 172.21.15.39:0/1290518146 conn(0x7f83240f7350 msgr2=0x7f83240f9480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:29.400 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.397+0000 7f832c2cb700 1 -- 172.21.15.39:0/1290518146 shutdown_connections 2023-06-22T17:26:29.400 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.397+0000 7f832c2cb700 1 -- 172.21.15.39:0/1290518146 wait complete. 2023-06-22T17:26:29.411 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-06-22T17:26:29.411 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-06-22T17:26:29.541 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.538+0000 7f32de47b700 1 -- 172.21.15.39:0/2142164802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f32d80fcc10 msgr2=0x7f32d80fd060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:29.541 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.538+0000 7f32de47b700 1 --2- 172.21.15.39:0/2142164802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f32d80fcc10 0x7f32d80fd060 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f32c80049c0 tx=0x7f32c8009e30 comp rx=0 tx=0).stop 2023-06-22T17:26:29.542 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.538+0000 7f32de47b700 1 -- 172.21.15.39:0/2142164802 shutdown_connections 2023-06-22T17:26:29.542 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.538+0000 7f32de47b700 1 --2- 172.21.15.39:0/2142164802 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f32d80fd5a0 0x7f32d8103e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:29.542 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.538+0000 7f32de47b700 1 --2- 172.21.15.39:0/2142164802 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f32d80fba00 0x7f32d80fbe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:29.542 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.538+0000 7f32de47b700 1 --2- 172.21.15.39:0/2142164802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f32d80fcc10 0x7f32d80fd060 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:29.542 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.538+0000 7f32de47b700 1 -- 172.21.15.39:0/2142164802 >> 172.21.15.39:0/2142164802 conn(0x7f32d80f7160 msgr2=0x7f32d80f9590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:29.542 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.538+0000 7f32de47b700 1 -- 172.21.15.39:0/2142164802 shutdown_connections 2023-06-22T17:26:29.542 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.538+0000 7f32de47b700 1 -- 172.21.15.39:0/2142164802 wait complete. 2023-06-22T17:26:29.543 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.538+0000 7f32de47b700 1 Processor -- start 2023-06-22T17:26:29.543 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.538+0000 7f32de47b700 1 -- start start 2023-06-22T17:26:29.543 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.538+0000 7f32de47b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f32d80fba00 0x7f32d810d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:29.543 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.538+0000 7f32de47b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f32d80fcc10 0x7f32d810ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:29.543 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.538+0000 7f32de47b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f32d80fd5a0 0x7f32d8114e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:29.543 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.539+0000 7f32de47b700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d8115450 con 0x7f32d80fcc10 2023-06-22T17:26:29.543 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.539+0000 7f32de47b700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d81155c0 con 0x7f32d80fd5a0 2023-06-22T17:26:29.543 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.539+0000 7f32de47b700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d8115780 con 0x7f32d80fba00 2023-06-22T17:26:29.544 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.539+0000 7f32dcc78700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f32d80fcc10 0x7f32d810ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:29.544 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.539+0000 7f32dcc78700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f32d80fcc10 0x7f32d810ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37328/0 (socket says 172.21.15.39:37328) 2023-06-22T17:26:29.544 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.539+0000 7f32dcc78700 1 -- 172.21.15.39:0/259485405 learned_addr learned my addr 172.21.15.39:0/259485405 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:26:29.544 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.539+0000 7f32ddc7a700 1 --2- 172.21.15.39:0/259485405 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f32d80fd5a0 0x7f32d8114e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:29.544 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.539+0000 7f32dd479700 1 --2- 172.21.15.39:0/259485405 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f32d80fba00 0x7f32d810d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:29.544 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.539+0000 7f32dcc78700 1 -- 172.21.15.39:0/259485405 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f32d80fba00 msgr2=0x7f32d810d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:29.544 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.539+0000 7f32dcc78700 1 --2- 172.21.15.39:0/259485405 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f32d80fba00 0x7f32d810d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:29.544 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.539+0000 7f32dcc78700 1 -- 172.21.15.39:0/259485405 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f32d80fd5a0 msgr2=0x7f32d8114e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:29.545 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.539+0000 7f32dcc78700 1 --2- 172.21.15.39:0/259485405 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f32d80fd5a0 0x7f32d8114e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:29.545 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.539+0000 7f32dcc78700 1 -- 172.21.15.39:0/259485405 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c8005040 con 0x7f32d80fcc10 2023-06-22T17:26:29.545 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.539+0000 7f32dd479700 1 --2- 172.21.15.39:0/259485405 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f32d80fba00 0x7f32d810d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-22T17:26:29.545 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.539+0000 7f32dcc78700 1 --2- 172.21.15.39:0/259485405 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f32d80fcc10 0x7f32d810ddb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f32c800bf20 tx=0x7f32c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:29.545 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.539+0000 7f32cdffb700 1 -- 172.21.15.39:0/259485405 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c800e040 con 0x7f32d80fcc10 2023-06-22T17:26:29.545 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.539+0000 7f32cdffb700 1 -- 172.21.15.39:0/259485405 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f32c8009830 con 0x7f32d80fcc10 2023-06-22T17:26:29.545 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.539+0000 7f32de47b700 1 -- 172.21.15.39:0/259485405 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32d8115a00 con 0x7f32d80fcc10 2023-06-22T17:26:29.546 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.539+0000 7f32cdffb700 1 -- 172.21.15.39:0/259485405 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c801c3f0 con 0x7f32d80fcc10 2023-06-22T17:26:29.546 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.540+0000 7f32de47b700 1 -- 172.21.15.39:0/259485405 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d8115ee0 con 0x7f32d80fcc10 2023-06-22T17:26:29.547 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.541+0000 7f32de47b700 1 -- 172.21.15.39:0/259485405 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32d8103a20 con 0x7f32d80fcc10 2023-06-22T17:26:29.547 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.541+0000 7f32cdffb700 1 -- 172.21.15.39:0/259485405 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f32c8009350 con 0x7f32d80fcc10 2023-06-22T17:26:29.547 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.542+0000 7f32cdffb700 1 --2- 172.21.15.39:0/259485405 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f32c409b680 0x7f32c409db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:29.547 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.542+0000 7f32cdffb700 1 -- 172.21.15.39:0/259485405 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f32c80c5c90 con 0x7f32d80fcc10 2023-06-22T17:26:29.547 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.542+0000 7f32dd479700 1 --2- 172.21.15.39:0/259485405 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f32c409b680 0x7f32c409db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:29.547 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.543+0000 7f32dd479700 1 --2- 172.21.15.39:0/259485405 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f32c409b680 0x7f32c409db30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f32d80fca70 tx=0x7f32d400b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:29.548 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.546+0000 7f32cdffb700 1 -- 172.21.15.39:0/259485405 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f32c8091990 con 0x7f32d80fcc10 2023-06-22T17:26:29.757 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:29.755+0000 7f32de47b700 1 -- 172.21.15.39:0/259485405 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f32d8107380 con 0x7f32d80fcc10 2023-06-22T17:26:30.626 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:30 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/4277973365' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-06-22T17:26:30.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:30 smithi101 ceph-mon[85507]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:30.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:30 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/1290518146' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-06-22T17:26:30.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:30 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:30.627 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:30 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:30.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:30 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/4277973365' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-06-22T17:26:30.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:30 smithi133 ceph-mon[86268]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:30.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:30 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/1290518146' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-06-22T17:26:30.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:30 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:30.672 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:30 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:30.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:30 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/4277973365' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-06-22T17:26:30.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:30 smithi039 ceph-mon[84805]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:30.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:30 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/1290518146' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-06-22T17:26:30.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:30 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:30.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:30 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:30.755 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.753+0000 7f32cdffb700 1 -- 172.21.15.39:0/259485405 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v78) v1 ==== 138+0+0 (secure 0 0 0) 0x7f32c8094fb0 con 0x7f32d80fcc10 2023-06-22T17:26:30.759 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.757+0000 7f32de47b700 1 -- 172.21.15.39:0/259485405 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f32c409b680 msgr2=0x7f32c409db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:30.759 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.757+0000 7f32de47b700 1 --2- 172.21.15.39:0/259485405 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f32c409b680 0x7f32c409db30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f32d80fca70 tx=0x7f32d400b040 comp rx=0 tx=0).stop 2023-06-22T17:26:30.759 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.757+0000 7f32de47b700 1 -- 172.21.15.39:0/259485405 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f32d80fcc10 msgr2=0x7f32d810ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:30.759 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.757+0000 7f32de47b700 1 --2- 172.21.15.39:0/259485405 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f32d80fcc10 0x7f32d810ddb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f32c800bf20 tx=0x7f32c8009e30 comp rx=0 tx=0).stop 2023-06-22T17:26:30.759 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.757+0000 7f32de47b700 1 -- 172.21.15.39:0/259485405 shutdown_connections 2023-06-22T17:26:30.760 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.757+0000 7f32de47b700 1 --2- 172.21.15.39:0/259485405 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f32c409b680 0x7f32c409db30 secure :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f32d80fca70 tx=0x7f32d400b040 comp rx=0 tx=0).stop 2023-06-22T17:26:30.760 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.757+0000 7f32de47b700 1 --2- 172.21.15.39:0/259485405 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f32d80fd5a0 0x7f32d8114e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:30.760 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.757+0000 7f32de47b700 1 --2- 172.21.15.39:0/259485405 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f32d80fba00 0x7f32d810d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:30.760 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.757+0000 7f32de47b700 1 --2- 172.21.15.39:0/259485405 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f32d80fcc10 0x7f32d810ddb0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:30.760 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.757+0000 7f32de47b700 1 -- 172.21.15.39:0/259485405 >> 172.21.15.39:0/259485405 conn(0x7f32d80f7160 msgr2=0x7f32d80f92f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:30.761 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.757+0000 7f32de47b700 1 -- 172.21.15.39:0/259485405 shutdown_connections 2023-06-22T17:26:30.761 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.757+0000 7f32de47b700 1 -- 172.21.15.39:0/259485405 wait complete. 2023-06-22T17:26:30.761 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_metadata' created 2023-06-22T17:26:30.772 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-06-22T17:26:30.907 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.899+0000 7f05c1652700 1 -- 172.21.15.39:0/3507728482 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05bc0fd780 msgr2=0x7f05bc104040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:30.907 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.899+0000 7f05c1652700 1 --2- 172.21.15.39:0/3507728482 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05bc0fd780 0x7f05bc104040 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f05b4009d60 tx=0x7f05b4004d10 comp rx=0 tx=0).stop 2023-06-22T17:26:30.908 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.899+0000 7f05c1652700 1 -- 172.21.15.39:0/3507728482 shutdown_connections 2023-06-22T17:26:30.908 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.899+0000 7f05c1652700 1 --2- 172.21.15.39:0/3507728482 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05bc0fd780 0x7f05bc104040 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:30.908 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.899+0000 7f05c1652700 1 --2- 172.21.15.39:0/3507728482 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f05bc0fbbe0 0x7f05bc0fbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:30.908 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.899+0000 7f05c1652700 1 --2- 172.21.15.39:0/3507728482 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05bc0fcdf0 0x7f05bc0fd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:30.908 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.899+0000 7f05c1652700 1 -- 172.21.15.39:0/3507728482 >> 172.21.15.39:0/3507728482 conn(0x7f05bc0f7320 msgr2=0x7f05bc0f9770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:30.908 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.899+0000 7f05c1652700 1 -- 172.21.15.39:0/3507728482 shutdown_connections 2023-06-22T17:26:30.909 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.899+0000 7f05c1652700 1 -- 172.21.15.39:0/3507728482 wait complete. 2023-06-22T17:26:30.909 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.899+0000 7f05c1652700 1 Processor -- start 2023-06-22T17:26:30.909 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.899+0000 7f05c1652700 1 -- start start 2023-06-22T17:26:30.909 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.900+0000 7f05c1652700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f05bc0fbbe0 0x7f05bc10b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:30.909 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.900+0000 7f05c1652700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05bc0fcdf0 0x7f05bc10bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:30.909 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.900+0000 7f05c1652700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05bc0fd780 0x7f05bc112bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:30.909 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.900+0000 7f05c1652700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc1131e0 con 0x7f05bc0fd780 2023-06-22T17:26:30.910 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.900+0000 7f05c1652700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc113350 con 0x7f05bc0fbbe0 2023-06-22T17:26:30.910 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.900+0000 7f05c1652700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc113510 con 0x7f05bc0fcdf0 2023-06-22T17:26:30.910 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.900+0000 7f05bb7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05bc0fd780 0x7f05bc112bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:30.910 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.900+0000 7f05baffd700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f05bc0fbbe0 0x7f05bc10b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:30.910 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.900+0000 7f05baffd700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f05bc0fbbe0 0x7f05bc10b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.39:55734/0 (socket says 172.21.15.39:55734) 2023-06-22T17:26:30.910 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.900+0000 7f05baffd700 1 -- 172.21.15.39:0/3178758939 learned_addr learned my addr 172.21.15.39:0/3178758939 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:26:30.910 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.900+0000 7f05bb7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05bc0fd780 0x7f05bc112bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37354/0 (socket says 172.21.15.39:37354) 2023-06-22T17:26:30.911 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.900+0000 7f05ba7fc700 1 --2- 172.21.15.39:0/3178758939 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05bc0fcdf0 0x7f05bc10bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:30.911 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.900+0000 7f05bb7fe700 1 -- 172.21.15.39:0/3178758939 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05bc0fcdf0 msgr2=0x7f05bc10bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:30.911 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.900+0000 7f05bb7fe700 1 --2- 172.21.15.39:0/3178758939 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05bc0fcdf0 0x7f05bc10bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:30.911 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.900+0000 7f05bb7fe700 1 -- 172.21.15.39:0/3178758939 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f05bc0fbbe0 msgr2=0x7f05bc10b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:30.911 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.900+0000 7f05bb7fe700 1 --2- 172.21.15.39:0/3178758939 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f05bc0fbbe0 0x7f05bc10b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:30.911 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.900+0000 7f05bb7fe700 1 -- 172.21.15.39:0/3178758939 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b0005040 con 0x7f05bc0fd780 2023-06-22T17:26:30.911 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.900+0000 7f05baffd700 1 --2- 172.21.15.39:0/3178758939 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f05bc0fbbe0 0x7f05bc10b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T17:26:30.912 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.900+0000 7f05ba7fc700 1 --2- 172.21.15.39:0/3178758939 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05bc0fcdf0 0x7f05bc10bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T17:26:30.912 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.900+0000 7f05bb7fe700 1 --2- 172.21.15.39:0/3178758939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05bc0fd780 0x7f05bc112bd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f05b400b690 tx=0x7f05b4004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:30.912 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.901+0000 7f05a37fe700 1 -- 172.21.15.39:0/3178758939 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b400e040 con 0x7f05bc0fd780 2023-06-22T17:26:30.912 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.901+0000 7f05a37fe700 1 -- 172.21.15.39:0/3178758939 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f05b401cb00 con 0x7f05bc0fd780 2023-06-22T17:26:30.912 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.901+0000 7f05c1652700 1 -- 172.21.15.39:0/3178758939 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05b4009940 con 0x7f05bc0fd780 2023-06-22T17:26:30.912 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.901+0000 7f05a37fe700 1 -- 172.21.15.39:0/3178758939 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b4012680 con 0x7f05bc0fd780 2023-06-22T17:26:30.912 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.901+0000 7f05c1652700 1 -- 172.21.15.39:0/3178758939 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05bc113a90 con 0x7f05bc0fd780 2023-06-22T17:26:30.913 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.902+0000 7f05c1652700 1 -- 172.21.15.39:0/3178758939 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05bc102ea0 con 0x7f05bc0fd780 2023-06-22T17:26:30.913 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.903+0000 7f05a37fe700 1 -- 172.21.15.39:0/3178758939 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f05b4019050 con 0x7f05bc0fd780 2023-06-22T17:26:30.913 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.904+0000 7f05a37fe700 1 --2- 172.21.15.39:0/3178758939 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f05a409b730 0x7f05a409dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:30.913 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.904+0000 7f05baffd700 1 --2- 172.21.15.39:0/3178758939 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f05a409b730 0x7f05a409dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:30.913 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.904+0000 7f05a37fe700 1 -- 172.21.15.39:0/3178758939 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7493+0+0 (secure 0 0 0) 0x7f05b40c6b10 con 0x7f05bc0fd780 2023-06-22T17:26:30.914 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.904+0000 7f05baffd700 1 --2- 172.21.15.39:0/3178758939 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f05a409b730 0x7f05a409dbe0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f05b00041d0 tx=0x7f05b000b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:30.914 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:30.907+0000 7f05a37fe700 1 -- 172.21.15.39:0/3178758939 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f05b40926a0 con 0x7f05bc0fd780 2023-06-22T17:26:31.105 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.103+0000 7f05c1652700 1 -- 172.21.15.39:0/3178758939 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f05bc106440 con 0x7f05bc0fd780 2023-06-22T17:26:31.767 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.764+0000 7f05a37fe700 1 -- 172.21.15.39:0/3178758939 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v79) v1 ==== 170+0+0 (secure 0 0 0) 0x7f05b4004810 con 0x7f05bc0fd780 2023-06-22T17:26:31.779 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.777+0000 7f05c1652700 1 -- 172.21.15.39:0/3178758939 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f05a409b730 msgr2=0x7f05a409dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:31.779 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.777+0000 7f05c1652700 1 --2- 172.21.15.39:0/3178758939 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f05a409b730 0x7f05a409dbe0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f05b00041d0 tx=0x7f05b000b040 comp rx=0 tx=0).stop 2023-06-22T17:26:31.779 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.777+0000 7f05c1652700 1 -- 172.21.15.39:0/3178758939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05bc0fd780 msgr2=0x7f05bc112bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:31.779 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.777+0000 7f05c1652700 1 --2- 172.21.15.39:0/3178758939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05bc0fd780 0x7f05bc112bd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f05b400b690 tx=0x7f05b4004d10 comp rx=0 tx=0).stop 2023-06-22T17:26:31.780 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.778+0000 7f05c1652700 1 -- 172.21.15.39:0/3178758939 shutdown_connections 2023-06-22T17:26:31.780 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.778+0000 7f05c1652700 1 --2- 172.21.15.39:0/3178758939 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f05a409b730 0x7f05a409dbe0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:31.780 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.778+0000 7f05c1652700 1 --2- 172.21.15.39:0/3178758939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05bc0fd780 0x7f05bc112bd0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:31.780 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.778+0000 7f05c1652700 1 --2- 172.21.15.39:0/3178758939 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f05bc0fbbe0 0x7f05bc10b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:31.780 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.778+0000 7f05c1652700 1 --2- 172.21.15.39:0/3178758939 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05bc0fcdf0 0x7f05bc10bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:31.781 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.778+0000 7f05c1652700 1 -- 172.21.15.39:0/3178758939 >> 172.21.15.39:0/3178758939 conn(0x7f05bc0f7320 msgr2=0x7f05bc100020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:31.781 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.778+0000 7f05c1652700 1 -- 172.21.15.39:0/3178758939 shutdown_connections 2023-06-22T17:26:31.781 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.778+0000 7f05c1652700 1 -- 172.21.15.39:0/3178758939 wait complete. 2023-06-22T17:26:31.781 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_data' created 2023-06-22T17:26:31.792 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-06-22T17:26:31.814 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:31 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/259485405' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-06-22T17:26:31.814 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:31 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:31.814 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:31 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:31.815 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:31 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:31.815 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:31 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:31.815 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:31 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/259485405' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-06-22T17:26:31.815 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:31 smithi039 ceph-mon[84805]: osdmap e78: 12 total, 12 up, 12 in 2023-06-22T17:26:31.815 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:31 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:31.815 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:31 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:26:31.815 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:31 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:31.815 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:31 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:26:31.876 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:31 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/259485405' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-06-22T17:26:31.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:31 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:31.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:31 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:31.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:31 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:31.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:31 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:31.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:31 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/259485405' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-06-22T17:26:31.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:31 smithi101 ceph-mon[85507]: osdmap e78: 12 total, 12 up, 12 in 2023-06-22T17:26:31.878 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:31 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:31.878 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:31 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:26:31.878 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:31 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:31.878 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:31 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:26:31.920 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:31 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/259485405' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-06-22T17:26:31.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:31 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:31.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:31 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:31.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:31 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:31.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:31 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:31.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:31 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/259485405' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-06-22T17:26:31.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:31 smithi133 ceph-mon[86268]: osdmap e78: 12 total, 12 up, 12 in 2023-06-22T17:26:31.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:31 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-22T17:26:31.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:31 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-22T17:26:31.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:31 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:31.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:31 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-22T17:26:31.923 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.921+0000 7f77dc224700 1 -- 172.21.15.39:0/3989882810 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f77d40fce30 msgr2=0x7f77d40fd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:31.923 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.921+0000 7f77dc224700 1 --2- 172.21.15.39:0/3989882810 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f77d40fce30 0x7f77d40fd280 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f77c40049c0 tx=0x7f77c4009e30 comp rx=0 tx=0).stop 2023-06-22T17:26:31.923 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.921+0000 7f77dc224700 1 -- 172.21.15.39:0/3989882810 shutdown_connections 2023-06-22T17:26:31.923 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.921+0000 7f77dc224700 1 --2- 172.21.15.39:0/3989882810 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f77d40fd7c0 0x7f77d4104080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:31.923 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.921+0000 7f77dc224700 1 --2- 172.21.15.39:0/3989882810 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f77d40fbc20 0x7f77d40fc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:31.924 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.921+0000 7f77dc224700 1 --2- 172.21.15.39:0/3989882810 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f77d40fce30 0x7f77d40fd280 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:31.924 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.921+0000 7f77dc224700 1 -- 172.21.15.39:0/3989882810 >> 172.21.15.39:0/3989882810 conn(0x7f77d40f7380 msgr2=0x7f77d40f97b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:31.924 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.921+0000 7f77dc224700 1 -- 172.21.15.39:0/3989882810 shutdown_connections 2023-06-22T17:26:31.924 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.921+0000 7f77dc224700 1 -- 172.21.15.39:0/3989882810 wait complete. 2023-06-22T17:26:31.925 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.921+0000 7f77dc224700 1 Processor -- start 2023-06-22T17:26:31.926 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.921+0000 7f77dc224700 1 -- start start 2023-06-22T17:26:31.926 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.921+0000 7f77dc224700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f77d40fbc20 0x7f77d410d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:31.926 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.921+0000 7f77dc224700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f77d40fce30 0x7f77d410ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:31.926 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.922+0000 7f77dc224700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f77d40fd7c0 0x7f77d4114f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:31.926 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.922+0000 7f77dc224700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d4115570 con 0x7f77d40fce30 2023-06-22T17:26:31.926 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.922+0000 7f77dc224700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d41156e0 con 0x7f77d40fbc20 2023-06-22T17:26:31.927 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.922+0000 7f77dc224700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d41158a0 con 0x7f77d40fd7c0 2023-06-22T17:26:31.927 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.922+0000 7f77d97bf700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f77d40fce30 0x7f77d410ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:31.927 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.922+0000 7f77d97bf700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f77d40fce30 0x7f77d410ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37368/0 (socket says 172.21.15.39:37368) 2023-06-22T17:26:31.927 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.922+0000 7f77d97bf700 1 -- 172.21.15.39:0/3418794032 learned_addr learned my addr 172.21.15.39:0/3418794032 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:26:31.927 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.922+0000 7f77d9fc0700 1 --2- 172.21.15.39:0/3418794032 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f77d40fbc20 0x7f77d410d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:31.927 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.922+0000 7f77da7c1700 1 --2- 172.21.15.39:0/3418794032 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f77d40fd7c0 0x7f77d4114f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:31.927 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.922+0000 7f77d97bf700 1 -- 172.21.15.39:0/3418794032 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f77d40fd7c0 msgr2=0x7f77d4114f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:31.927 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.922+0000 7f77d97bf700 1 --2- 172.21.15.39:0/3418794032 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f77d40fd7c0 0x7f77d4114f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:31.928 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.922+0000 7f77d97bf700 1 -- 172.21.15.39:0/3418794032 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f77d40fbc20 msgr2=0x7f77d410d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:31.928 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.922+0000 7f77d97bf700 1 --2- 172.21.15.39:0/3418794032 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f77d40fbc20 0x7f77d410d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:31.928 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.922+0000 7f77d97bf700 1 -- 172.21.15.39:0/3418794032 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c4005040 con 0x7f77d40fce30 2023-06-22T17:26:31.928 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.922+0000 7f77da7c1700 1 --2- 172.21.15.39:0/3418794032 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f77d40fd7c0 0x7f77d4114f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-22T17:26:31.928 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.922+0000 7f77d97bf700 1 --2- 172.21.15.39:0/3418794032 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f77d40fce30 0x7f77d410ded0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f77c4009f00 tx=0x7f77c4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:31.928 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.922+0000 7f77ca7fc700 1 -- 172.21.15.39:0/3418794032 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c400e040 con 0x7f77d40fce30 2023-06-22T17:26:31.928 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.922+0000 7f77ca7fc700 1 -- 172.21.15.39:0/3418794032 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f77c4013420 con 0x7f77d40fce30 2023-06-22T17:26:31.928 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.922+0000 7f77ca7fc700 1 -- 172.21.15.39:0/3418794032 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c401c620 con 0x7f77d40fce30 2023-06-22T17:26:31.929 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.922+0000 7f77dc224700 1 -- 172.21.15.39:0/3418794032 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77d4115b20 con 0x7f77d40fce30 2023-06-22T17:26:31.929 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.922+0000 7f77dc224700 1 -- 172.21.15.39:0/3418794032 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77d4115f30 con 0x7f77d40fce30 2023-06-22T17:26:31.929 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.923+0000 7f77dc224700 1 -- 172.21.15.39:0/3418794032 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77d4103ab0 con 0x7f77d40fce30 2023-06-22T17:26:31.929 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.924+0000 7f77ca7fc700 1 -- 172.21.15.39:0/3418794032 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f77c4013590 con 0x7f77d40fce30 2023-06-22T17:26:31.930 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.925+0000 7f77ca7fc700 1 --2- 172.21.15.39:0/3418794032 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f77c009b730 0x7f77c009dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:31.930 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.925+0000 7f77d9fc0700 1 --2- 172.21.15.39:0/3418794032 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f77c009b730 0x7f77c009dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:31.930 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.925+0000 7f77ca7fc700 1 -- 172.21.15.39:0/3418794032 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7875+0+0 (secure 0 0 0) 0x7f77c40c6850 con 0x7f77d40fce30 2023-06-22T17:26:31.930 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.925+0000 7f77d9fc0700 1 --2- 172.21.15.39:0/3418794032 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f77c009b730 0x7f77c009dbe0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f77d0009fd0 tx=0x7f77d000b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:31.930 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:31.928+0000 7f77ca7fc700 1 -- 172.21.15.39:0/3418794032 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f77c40921c0 con 0x7f77d40fce30 2023-06-22T17:26:32.174 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.172+0000 7f77dc224700 1 -- 172.21.15.39:0/3418794032 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f77d402b280 con 0x7f77d40fce30 2023-06-22T17:26:32.186 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.183+0000 7f77ca7fc700 1 -- 172.21.15.39:0/3418794032 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f77c40957e0 con 0x7f77d40fce30 2023-06-22T17:26:32.196 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.194+0000 7f77dc224700 1 -- 172.21.15.39:0/3418794032 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f77c009b730 msgr2=0x7f77c009dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:32.196 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.194+0000 7f77dc224700 1 --2- 172.21.15.39:0/3418794032 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f77c009b730 0x7f77c009dbe0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f77d0009fd0 tx=0x7f77d000b040 comp rx=0 tx=0).stop 2023-06-22T17:26:32.196 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.194+0000 7f77dc224700 1 -- 172.21.15.39:0/3418794032 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f77d40fce30 msgr2=0x7f77d410ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:32.196 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.194+0000 7f77dc224700 1 --2- 172.21.15.39:0/3418794032 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f77d40fce30 0x7f77d410ded0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f77c4009f00 tx=0x7f77c4009e30 comp rx=0 tx=0).stop 2023-06-22T17:26:32.197 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.194+0000 7f77dc224700 1 -- 172.21.15.39:0/3418794032 shutdown_connections 2023-06-22T17:26:32.197 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.194+0000 7f77dc224700 1 --2- 172.21.15.39:0/3418794032 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f77c009b730 0x7f77c009dbe0 secure :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f77d0009fd0 tx=0x7f77d000b040 comp rx=0 tx=0).stop 2023-06-22T17:26:32.197 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.194+0000 7f77dc224700 1 --2- 172.21.15.39:0/3418794032 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f77d40fd7c0 0x7f77d4114f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:32.197 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.194+0000 7f77dc224700 1 --2- 172.21.15.39:0/3418794032 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f77d40fbc20 0x7f77d410d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:32.197 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.194+0000 7f77dc224700 1 --2- 172.21.15.39:0/3418794032 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f77d40fce30 0x7f77d410ded0 secure :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f77c4009f00 tx=0x7f77c4009e30 comp rx=0 tx=0).stop 2023-06-22T17:26:32.197 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.194+0000 7f77dc224700 1 -- 172.21.15.39:0/3418794032 >> 172.21.15.39:0/3418794032 conn(0x7f77d40f7380 msgr2=0x7f77d40f9340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:32.198 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.197+0000 7f77dc224700 1 -- 172.21.15.39:0/3418794032 shutdown_connections 2023-06-22T17:26:32.199 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.197+0000 7f77dc224700 1 -- 172.21.15.39:0/3418794032 wait complete. 2023-06-22T17:26:32.208 INFO:teuthology.orchestra.run.smithi039.stderr:new fs with metadata pool 2 and data pool 3 2023-06-22T17:26:32.219 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-22T17:26:32.384 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.382+0000 7f3e70366700 1 -- 172.21.15.39:0/1038760008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e680ffe50 msgr2=0x7f3e681022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:32.384 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.382+0000 7f3e70366700 1 --2- 172.21.15.39:0/1038760008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e680ffe50 0x7f3e681022b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3e580049c0 tx=0x7f3e58009e30 comp rx=0 tx=0).stop 2023-06-22T17:26:32.385 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.382+0000 7f3e70366700 1 -- 172.21.15.39:0/1038760008 shutdown_connections 2023-06-22T17:26:32.385 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.382+0000 7f3e70366700 1 --2- 172.21.15.39:0/1038760008 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3e68102940 0x7f3e68104df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:32.385 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.382+0000 7f3e70366700 1 --2- 172.21.15.39:0/1038760008 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3e68065cb0 0x7f3e680ff880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:32.385 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.382+0000 7f3e70366700 1 --2- 172.21.15.39:0/1038760008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e680ffe50 0x7f3e681022b0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:32.385 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.382+0000 7f3e70366700 1 -- 172.21.15.39:0/1038760008 >> 172.21.15.39:0/1038760008 conn(0x7f3e680f7310 msgr2=0x7f3e680f9760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:32.385 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.382+0000 7f3e70366700 1 -- 172.21.15.39:0/1038760008 shutdown_connections 2023-06-22T17:26:32.385 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.382+0000 7f3e70366700 1 -- 172.21.15.39:0/1038760008 wait complete. 2023-06-22T17:26:32.385 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.382+0000 7f3e70366700 1 Processor -- start 2023-06-22T17:26:32.386 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.382+0000 7f3e70366700 1 -- start start 2023-06-22T17:26:32.387 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.383+0000 7f3e70366700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3e68065cb0 0x7f3e680fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:32.387 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.383+0000 7f3e70366700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3e680ffe50 0x7f3e680fb780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:32.388 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.383+0000 7f3e70366700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e68102940 0x7f3e680fbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:32.388 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.383+0000 7f3e70366700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e680fc430 con 0x7f3e68102940 2023-06-22T17:26:32.388 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.383+0000 7f3e70366700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e680fc5a0 con 0x7f3e68065cb0 2023-06-22T17:26:32.388 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.383+0000 7f3e70366700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e68117ae0 con 0x7f3e680ffe50 2023-06-22T17:26:32.388 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.383+0000 7f3e6e903700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e68102940 0x7f3e680fbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:32.388 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.383+0000 7f3e6e903700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e68102940 0x7f3e680fbcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37376/0 (socket says 172.21.15.39:37376) 2023-06-22T17:26:32.388 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.383+0000 7f3e6e903700 1 -- 172.21.15.39:0/3154074577 learned_addr learned my addr 172.21.15.39:0/3154074577 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:26:32.389 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.383+0000 7f3e6e102700 1 --2- 172.21.15.39:0/3154074577 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3e68065cb0 0x7f3e680fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:32.389 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.383+0000 7f3e6d901700 1 --2- 172.21.15.39:0/3154074577 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3e680ffe50 0x7f3e680fb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:32.389 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.383+0000 7f3e6e903700 1 -- 172.21.15.39:0/3154074577 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3e680ffe50 msgr2=0x7f3e680fb780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:32.389 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.383+0000 7f3e6e903700 1 --2- 172.21.15.39:0/3154074577 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3e680ffe50 0x7f3e680fb780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:32.389 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.383+0000 7f3e6e903700 1 -- 172.21.15.39:0/3154074577 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3e68065cb0 msgr2=0x7f3e680fd120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:32.389 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.383+0000 7f3e6e903700 1 --2- 172.21.15.39:0/3154074577 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3e68065cb0 0x7f3e680fd120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:32.389 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.383+0000 7f3e6e903700 1 -- 172.21.15.39:0/3154074577 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e58005040 con 0x7f3e68102940 2023-06-22T17:26:32.390 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.383+0000 7f3e6e903700 1 --2- 172.21.15.39:0/3154074577 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e68102940 0x7f3e680fbcc0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3e6000ec30 tx=0x7f3e6000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:32.390 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.383+0000 7f3e5effd700 1 -- 172.21.15.39:0/3154074577 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e60007b30 con 0x7f3e68102940 2023-06-22T17:26:32.390 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.383+0000 7f3e5effd700 1 -- 172.21.15.39:0/3154074577 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f3e6000f040 con 0x7f3e68102940 2023-06-22T17:26:32.390 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.383+0000 7f3e70366700 1 -- 172.21.15.39:0/3154074577 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e68117d90 con 0x7f3e68102940 2023-06-22T17:26:32.390 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.383+0000 7f3e5effd700 1 -- 172.21.15.39:0/3154074577 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6000cbb0 con 0x7f3e68102940 2023-06-22T17:26:32.390 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.383+0000 7f3e70366700 1 -- 172.21.15.39:0/3154074577 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e68118220 con 0x7f3e68102940 2023-06-22T17:26:32.390 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.384+0000 7f3e70366700 1 -- 172.21.15.39:0/3154074577 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e680f8e40 con 0x7f3e68102940 2023-06-22T17:26:32.390 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.385+0000 7f3e5effd700 1 -- 172.21.15.39:0/3154074577 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f3e60009090 con 0x7f3e68102940 2023-06-22T17:26:32.391 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.386+0000 7f3e5effd700 1 --2- 172.21.15.39:0/3154074577 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f3e5409b680 0x7f3e5409db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:32.391 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.386+0000 7f3e6e102700 1 --2- 172.21.15.39:0/3154074577 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f3e5409b680 0x7f3e5409db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:32.391 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.386+0000 7f3e5effd700 1 -- 172.21.15.39:0/3154074577 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f3e600c5770 con 0x7f3e68102940 2023-06-22T17:26:32.391 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.387+0000 7f3e6e102700 1 --2- 172.21.15.39:0/3154074577 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f3e5409b680 0x7f3e5409db30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3e64009fd0 tx=0x7f3e64009f60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:32.392 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.390+0000 7f3e5effd700 1 -- 172.21.15.39:0/3154074577 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f3e60091120 con 0x7f3e68102940 2023-06-22T17:26:32.597 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.595+0000 7f3e70366700 1 -- 172.21.15.39:0/3154074577 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e6802b410 con 0x7f3e68102940 2023-06-22T17:26:32.598 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.596+0000 7f3e5effd700 1 -- 172.21.15.39:0/3154074577 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+21396 (secure 0 0 0) 0x7f3e60094740 con 0x7f3e68102940 2023-06-22T17:26:32.598 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:26:32.599 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":80,"fsid":"d88002e2-1120-11ee-9b2b-001a4aab830c","created":"2023-06-22T17:19:36.313754+0000","modified":"2023-06-22T17:26:32.174224+0000","last_up_change":"2023-06-22T17:25:51.023671+0000","last_in_change":"2023-06-22T17:25:33.400238+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-22T17:22:35.797853+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-06-22T17:26:30.187897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-06-22T17:26:31.538388+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"eb8f5cba-9266-47ec-8af1-8e9edd6ec96a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":453150307},{"type":"v1","addr":"172.21.15.39:6803","nonce":453150307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":453150307},{"type":"v1","addr":"172.21.15.39:6805","nonce":453150307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":453150307},{"type":"v1","addr":"172.21.15.39:6809","nonce":453150307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":453150307},{"type":"v1","addr":"172.21.15.39:6807","nonce":453150307}]},"public_addr":"172.21.15.39:6803/453150307","cluster_addr":"172.21.15.39:6805/453150307","heartbeat_back_addr":"172.21.15.39:6809/453150307","heartbeat_front_addr":"172.21.15.39:6807/453150307","state":["exists","up"]},{"osd":1,"uuid":"8b8dbe3e-fd72-4f92-98a2-e212df5366c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6800","nonce":2708845767},{"type":"v1","addr":"172.21.15.101:6801","nonce":2708845767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2708845767},{"type":"v1","addr":"172.21.15.101:6803","nonce":2708845767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2708845767},{"type":"v1","addr":"172.21.15.101:6807","nonce":2708845767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2708845767},{"type":"v1","addr":"172.21.15.101:6805","nonce":2708845767}]},"public_addr":"172.21.15.101:6801/2708845767","cluster_addr":"172.21.15.101:6803/2708845767","heartbeat_back_addr":"172.21.15.101:6807/2708845767","heartbeat_front_addr":"172.21.15.101:6805/2708845767","state":["exists","up"]},{"osd":2,"uuid":"7b25ade6-b5e2-4194-92c7-3baf8c4e9560","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":3632902853},{"type":"v1","addr":"172.21.15.133:6801","nonce":3632902853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":3632902853},{"type":"v1","addr":"172.21.15.133:6803","nonce":3632902853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":3632902853},{"type":"v1","addr":"172.21.15.133:6807","nonce":3632902853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":3632902853},{"type":"v1","addr":"172.21.15.133:6805","nonce":3632902853}]},"public_addr":"172.21.15.133:6801/3632902853","cluster_addr":"172.21.15.133:6803/3632902853","heartbeat_back_addr":"172.21.15.133:6807/3632902853","heartbeat_front_addr":"172.21.15.133:6805/3632902853","state":["exists","up"]},{"osd":3,"uuid":"b91b9111-cb8f-4c51-85c6-96c0851d57d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1215819990},{"type":"v1","addr":"172.21.15.39:6811","nonce":1215819990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1215819990},{"type":"v1","addr":"172.21.15.39:6813","nonce":1215819990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":1215819990},{"type":"v1","addr":"172.21.15.39:6817","nonce":1215819990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1215819990},{"type":"v1","addr":"172.21.15.39:6815","nonce":1215819990}]},"public_addr":"172.21.15.39:6811/1215819990","cluster_addr":"172.21.15.39:6813/1215819990","heartbeat_back_addr":"172.21.15.39:6817/1215819990","heartbeat_front_addr":"172.21.15.39:6815/1215819990","state":["exists","up"]},{"osd":4,"uuid":"14f6d038-1e3c-41e0-9be6-3615099ac5fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":1371317064},{"type":"v1","addr":"172.21.15.101:6809","nonce":1371317064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":1371317064},{"type":"v1","addr":"172.21.15.101:6811","nonce":1371317064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":1371317064},{"type":"v1","addr":"172.21.15.101:6815","nonce":1371317064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":1371317064},{"type":"v1","addr":"172.21.15.101:6813","nonce":1371317064}]},"public_addr":"172.21.15.101:6809/1371317064","cluster_addr":"172.21.15.101:6811/1371317064","heartbeat_back_addr":"172.21.15.101:6815/1371317064","heartbeat_front_addr":"172.21.15.101:6813/1371317064","state":["exists","up"]},{"osd":5,"uuid":"6daaeb92-c73d-4c30-bb1d-18d5305a63a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":747181981},{"type":"v1","addr":"172.21.15.133:6809","nonce":747181981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":747181981},{"type":"v1","addr":"172.21.15.133:6811","nonce":747181981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":747181981},{"type":"v1","addr":"172.21.15.133:6815","nonce":747181981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":747181981},{"type":"v1","addr":"172.21.15.133:6813","nonce":747181981}]},"public_addr":"172.21.15.133:6809/747181981","cluster_addr":"172.21.15.133:6811/747181981","heartbeat_back_addr":"172.21.15.133:6815/747181981","heartbeat_front_addr":"172.21.15.133:6813/747181981","state":["exists","up"]},{"osd":6,"uuid":"21f46569-8123-4bdd-b3c5-4cc8bc271899","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1221216882},{"type":"v1","addr":"172.21.15.39:6819","nonce":1221216882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1221216882},{"type":"v1","addr":"172.21.15.39:6821","nonce":1221216882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1221216882},{"type":"v1","addr":"172.21.15.39:6825","nonce":1221216882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1221216882},{"type":"v1","addr":"172.21.15.39:6823","nonce":1221216882}]},"public_addr":"172.21.15.39:6819/1221216882","cluster_addr":"172.21.15.39:6821/1221216882","heartbeat_back_addr":"172.21.15.39:6825/1221216882","heartbeat_front_addr":"172.21.15.39:6823/1221216882","state":["exists","up"]},{"osd":7,"uuid":"7a1cbc1b-8cf0-4153-b8e2-3d3313bed2ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":1606613660},{"type":"v1","addr":"172.21.15.101:6817","nonce":1606613660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":1606613660},{"type":"v1","addr":"172.21.15.101:6819","nonce":1606613660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":1606613660},{"type":"v1","addr":"172.21.15.101:6823","nonce":1606613660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":1606613660},{"type":"v1","addr":"172.21.15.101:6821","nonce":1606613660}]},"public_addr":"172.21.15.101:6817/1606613660","cluster_addr":"172.21.15.101:6819/1606613660","heartbeat_back_addr":"172.21.15.101:6823/1606613660","heartbeat_front_addr":"172.21.15.101:6821/1606613660","state":["exists","up"]},{"osd":8,"uuid":"ab09136e-4326-4c02-ac59-3d2264e85b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":1363381164},{"type":"v1","addr":"172.21.15.133:6817","nonce":1363381164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":1363381164},{"type":"v1","addr":"172.21.15.133:6819","nonce":1363381164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":1363381164},{"type":"v1","addr":"172.21.15.133:6823","nonce":1363381164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":1363381164},{"type":"v1","addr":"172.21.15.133:6821","nonce":1363381164}]},"public_addr":"172.21.15.133:6817/1363381164","cluster_addr":"172.21.15.133:6819/1363381164","heartbeat_back_addr":"172.21.15.133:6823/1363381164","heartbeat_front_addr":"172.21.15.133:6821/1363381164","state":["exists","up"]},{"osd":9,"uuid":"ec8c1344-0478-4094-b544-3360054fd92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3010986736},{"type":"v1","addr":"172.21.15.39:6827","nonce":3010986736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3010986736},{"type":"v1","addr":"172.21.15.39:6829","nonce":3010986736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3010986736},{"type":"v1","addr":"172.21.15.39:6833","nonce":3010986736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3010986736},{"type":"v1","addr":"172.21.15.39:6831","nonce":3010986736}]},"public_addr":"172.21.15.39:6827/3010986736","cluster_addr":"172.21.15.39:6829/3010986736","heartbeat_back_addr":"172.21.15.39:6833/3010986736","heartbeat_front_addr":"172.21.15.39:6831/3010986736","state":["exists","up"]},{"osd":10,"uuid":"9da2943f-b947-49d6-bab1-bc3a95299996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3982067769},{"type":"v1","addr":"172.21.15.101:6825","nonce":3982067769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":3982067769},{"type":"v1","addr":"172.21.15.101:6827","nonce":3982067769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":3982067769},{"type":"v1","addr":"172.21.15.101:6831","nonce":3982067769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":3982067769},{"type":"v1","addr":"172.21.15.101:6829","nonce":3982067769}]},"public_addr":"172.21.15.101:6825/3982067769","cluster_addr":"172.21.15.101:6827/3982067769","heartbeat_back_addr":"172.21.15.101:6831/3982067769","heartbeat_front_addr":"172.21.15.101:6829/3982067769","state":["exists","up"]},{"osd":11,"uuid":"ee5d51b4-c9ad-4603-a967-c60f3c1f2cdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":1265049047},{"type":"v1","addr":"172.21.15.133:6825","nonce":1265049047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6826","nonce":1265049047},{"type":"v1","addr":"172.21.15.133:6827","nonce":1265049047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6830","nonce":1265049047},{"type":"v1","addr":"172.21.15.133:6831","nonce":1265049047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6828","nonce":1265049047},{"type":"v1","addr":"172.21.15.133:6829","nonce":1265049047}]},"public_addr":"172.21.15.133:6825/1265049047","cluster_addr":"172.21.15.133:6827/1265049047","heartbeat_back_addr":"172.21.15.133:6831/1265049047","heartbeat_front_addr":"172.21.15.133:6829/1265049047","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:21:49.604087+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:22:11.298409+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:22:32.913915+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:22:56.095316+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:23:17.505593+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:23:38.490450+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:24:01.498630+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:24:22.952502+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:24:44.480122+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:25:07.141899+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:25:28.109970+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:25:49.227776+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:6801/1350590065":"2023-06-23T17:20:27.281966+0000","172.21.15.39:6800/1350590065":"2023-06-23T17:20:27.281966+0000","172.21.15.39:0/82302526":"2023-06-23T17:20:27.281966+0000","172.21.15.39:0/260892351":"2023-06-23T17:20:05.206927+0000","172.21.15.39:0/1513096987":"2023-06-23T17:20:05.206927+0000","172.21.15.39:6800/3960151496":"2023-06-23T17:20:05.206927+0000","172.21.15.39:6801/3960151496":"2023-06-23T17:20:05.206927+0000","172.21.15.39:0/2480502158":"2023-06-23T17:20:27.281966+0000","172.21.15.39:0/3848722689":"2023-06-23T17:20:27.281966+0000","172.21.15.39:0/2257210574":"2023-06-23T17:20:05.206927+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T17:26:32.600 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.598+0000 7f3e70366700 1 -- 172.21.15.39:0/3154074577 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f3e5409b680 msgr2=0x7f3e5409db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:32.601 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.598+0000 7f3e70366700 1 --2- 172.21.15.39:0/3154074577 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f3e5409b680 0x7f3e5409db30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3e64009fd0 tx=0x7f3e64009f60 comp rx=0 tx=0).stop 2023-06-22T17:26:32.601 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.598+0000 7f3e70366700 1 -- 172.21.15.39:0/3154074577 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e68102940 msgr2=0x7f3e680fbcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:32.601 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.598+0000 7f3e70366700 1 --2- 172.21.15.39:0/3154074577 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e68102940 0x7f3e680fbcc0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3e6000ec30 tx=0x7f3e6000b040 comp rx=0 tx=0).stop 2023-06-22T17:26:32.601 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.598+0000 7f3e70366700 1 -- 172.21.15.39:0/3154074577 shutdown_connections 2023-06-22T17:26:32.601 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.598+0000 7f3e70366700 1 --2- 172.21.15.39:0/3154074577 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f3e5409b680 0x7f3e5409db30 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:32.601 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.598+0000 7f3e70366700 1 --2- 172.21.15.39:0/3154074577 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e68102940 0x7f3e680fbcc0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:32.601 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.598+0000 7f3e70366700 1 --2- 172.21.15.39:0/3154074577 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3e68065cb0 0x7f3e680fd120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:32.602 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.598+0000 7f3e70366700 1 --2- 172.21.15.39:0/3154074577 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3e680ffe50 0x7f3e680fb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:32.602 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.598+0000 7f3e70366700 1 -- 172.21.15.39:0/3154074577 >> 172.21.15.39:0/3154074577 conn(0x7f3e680f7310 msgr2=0x7f3e680fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:32.602 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.599+0000 7f3e70366700 1 -- 172.21.15.39:0/3154074577 shutdown_connections 2023-06-22T17:26:32.602 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.599+0000 7f3e70366700 1 -- 172.21.15.39:0/3154074577 wait complete. 2023-06-22T17:26:32.614 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-22T17:26:32.737 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.735+0000 7f64af59e700 1 -- 172.21.15.39:0/4234730426 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f64b00fb9e0 msgr2=0x7f64b00fbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:32.737 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.735+0000 7f64af59e700 1 --2- 172.21.15.39:0/4234730426 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f64b00fb9e0 0x7f64b00fbdf0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f64a00049c0 tx=0x7f64a0009e30 comp rx=0 tx=0).stop 2023-06-22T17:26:32.737 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.735+0000 7f64af59e700 1 -- 172.21.15.39:0/4234730426 shutdown_connections 2023-06-22T17:26:32.737 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.735+0000 7f64af59e700 1 --2- 172.21.15.39:0/4234730426 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f64b00fd640 0x7f64b0103ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:32.738 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.735+0000 7f64af59e700 1 --2- 172.21.15.39:0/4234730426 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f64b00fb9e0 0x7f64b00fbdf0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:32.738 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.735+0000 7f64af59e700 1 --2- 172.21.15.39:0/4234730426 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f64b00fcc90 0x7f64b00fd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:32.738 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.735+0000 7f64af59e700 1 -- 172.21.15.39:0/4234730426 >> 172.21.15.39:0/4234730426 conn(0x7f64b00f7140 msgr2=0x7f64b00f9570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:32.738 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.735+0000 7f64af59e700 1 -- 172.21.15.39:0/4234730426 shutdown_connections 2023-06-22T17:26:32.738 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.735+0000 7f64af59e700 1 -- 172.21.15.39:0/4234730426 wait complete. 2023-06-22T17:26:32.739 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.735+0000 7f64af59e700 1 Processor -- start 2023-06-22T17:26:32.740 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.735+0000 7f64af59e700 1 -- start start 2023-06-22T17:26:32.740 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.736+0000 7f64af59e700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f64b00fb9e0 0x7f64b0065a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:32.740 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.736+0000 7f64af59e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f64b00fcc90 0x7f64b01125d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:32.740 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.736+0000 7f64af59e700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f64b00fd640 0x7f64b0114bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:32.740 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.736+0000 7f64af59e700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b01153e0 con 0x7f64b00fcc90 2023-06-22T17:26:32.740 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.736+0000 7f64af59e700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b0115550 con 0x7f64b00fd640 2023-06-22T17:26:32.741 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.736+0000 7f64af59e700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b0115710 con 0x7f64b00fb9e0 2023-06-22T17:26:32.741 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.736+0000 7f64add9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f64b00fcc90 0x7f64b01125d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:32.741 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.736+0000 7f64add9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f64b00fcc90 0x7f64b01125d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37406/0 (socket says 172.21.15.39:37406) 2023-06-22T17:26:32.741 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.736+0000 7f64add9b700 1 -- 172.21.15.39:0/3463376769 learned_addr learned my addr 172.21.15.39:0/3463376769 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:26:32.741 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.736+0000 7f64aed9d700 1 --2- 172.21.15.39:0/3463376769 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f64b00fd640 0x7f64b0114bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:32.741 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.736+0000 7f64ae59c700 1 --2- 172.21.15.39:0/3463376769 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f64b00fb9e0 0x7f64b0065a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:32.741 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.736+0000 7f64add9b700 1 -- 172.21.15.39:0/3463376769 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f64b00fb9e0 msgr2=0x7f64b0065a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:32.741 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.736+0000 7f64add9b700 1 --2- 172.21.15.39:0/3463376769 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f64b00fb9e0 0x7f64b0065a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:32.742 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.736+0000 7f64add9b700 1 -- 172.21.15.39:0/3463376769 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f64b00fd640 msgr2=0x7f64b0114bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:32.742 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.736+0000 7f64add9b700 1 --2- 172.21.15.39:0/3463376769 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f64b00fd640 0x7f64b0114bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:32.742 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.736+0000 7f64add9b700 1 -- 172.21.15.39:0/3463376769 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a0005040 con 0x7f64b00fcc90 2023-06-22T17:26:32.742 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.736+0000 7f64add9b700 1 --2- 172.21.15.39:0/3463376769 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f64b00fcc90 0x7f64b01125d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f64a400ebc0 tx=0x7f64a400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:32.742 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.736+0000 7f649effd700 1 -- 172.21.15.39:0/3463376769 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a4007cc0 con 0x7f64b00fcc90 2023-06-22T17:26:32.742 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.736+0000 7f64af59e700 1 -- 172.21.15.39:0/3463376769 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64b0115930 con 0x7f64b00fcc90 2023-06-22T17:26:32.742 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.736+0000 7f649effd700 1 -- 172.21.15.39:0/3463376769 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f64a400f040 con 0x7f64b00fcc90 2023-06-22T17:26:32.742 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.736+0000 7f649effd700 1 -- 172.21.15.39:0/3463376769 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a4009480 con 0x7f64b00fcc90 2023-06-22T17:26:32.743 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.737+0000 7f64af59e700 1 -- 172.21.15.39:0/3463376769 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64b0115e70 con 0x7f64b00fcc90 2023-06-22T17:26:32.743 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.737+0000 7f64af59e700 1 -- 172.21.15.39:0/3463376769 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64b0102800 con 0x7f64b00fcc90 2023-06-22T17:26:32.743 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.739+0000 7f649effd700 1 -- 172.21.15.39:0/3463376769 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f64a400c600 con 0x7f64b00fcc90 2023-06-22T17:26:32.744 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.740+0000 7f649effd700 1 --2- 172.21.15.39:0/3463376769 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f649809b680 0x7f649809db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:32.744 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.740+0000 7f64ae59c700 1 --2- 172.21.15.39:0/3463376769 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f649809b680 0x7f649809db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:32.744 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.740+0000 7f649effd700 1 -- 172.21.15.39:0/3463376769 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f64a40c4d30 con 0x7f64b00fcc90 2023-06-22T17:26:32.744 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.740+0000 7f64ae59c700 1 --2- 172.21.15.39:0/3463376769 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f649809b680 0x7f649809db30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f64a0005010 tx=0x7f64a0009e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:32.745 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.743+0000 7f649effd700 1 -- 172.21.15.39:0/3463376769 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f64a40906c0 con 0x7f64b00fcc90 2023-06-22T17:26:32.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:32 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/3178758939' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-06-22T17:26:32.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:32 smithi101 ceph-mon[85507]: pgmap v250: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:32.877 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:32 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/3178758939' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-06-22T17:26:32.878 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:32 smithi101 ceph-mon[85507]: osdmap e79: 12 total, 12 up, 12 in 2023-06-22T17:26:32.878 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:32 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/3418794032' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-06-22T17:26:32.878 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:32 smithi101 ceph-mon[85507]: daemon mds.i assigned to filesystem cephfs as rank 0 2023-06-22T17:26:32.879 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:32 smithi101 ceph-mon[85507]: osdmap e80: 12 total, 12 up, 12 in 2023-06-22T17:26:32.879 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:32 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/3418794032' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-06-22T17:26:32.879 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:32 smithi101 ceph-mon[85507]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2023-06-22T17:26:32.879 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:32 smithi101 ceph-mon[85507]: daemon mds.i is now active in filesystem cephfs as rank 0 2023-06-22T17:26:32.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:32 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/3178758939' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-06-22T17:26:32.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:32 smithi133 ceph-mon[86268]: pgmap v250: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:32.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:32 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/3178758939' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-06-22T17:26:32.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:32 smithi133 ceph-mon[86268]: osdmap e79: 12 total, 12 up, 12 in 2023-06-22T17:26:32.921 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:32 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/3418794032' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-06-22T17:26:32.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:32 smithi133 ceph-mon[86268]: daemon mds.i assigned to filesystem cephfs as rank 0 2023-06-22T17:26:32.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:32 smithi133 ceph-mon[86268]: osdmap e80: 12 total, 12 up, 12 in 2023-06-22T17:26:32.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:32 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/3418794032' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-06-22T17:26:32.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:32 smithi133 ceph-mon[86268]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2023-06-22T17:26:32.922 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:32 smithi133 ceph-mon[86268]: daemon mds.i is now active in filesystem cephfs as rank 0 2023-06-22T17:26:32.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:32 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/3178758939' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-06-22T17:26:32.933 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:32 smithi039 ceph-mon[84805]: pgmap v250: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:32.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:32 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/3178758939' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-06-22T17:26:32.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:32 smithi039 ceph-mon[84805]: osdmap e79: 12 total, 12 up, 12 in 2023-06-22T17:26:32.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:32 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/3418794032' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-06-22T17:26:32.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:32 smithi039 ceph-mon[84805]: daemon mds.i assigned to filesystem cephfs as rank 0 2023-06-22T17:26:32.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:32 smithi039 ceph-mon[84805]: osdmap e80: 12 total, 12 up, 12 in 2023-06-22T17:26:32.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:32 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/3418794032' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-06-22T17:26:32.934 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:32 smithi039 ceph-mon[84805]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2023-06-22T17:26:32.935 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:32 smithi039 ceph-mon[84805]: daemon mds.i is now active in filesystem cephfs as rank 0 2023-06-22T17:26:32.947 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.945+0000 7f64af59e700 1 -- 172.21.15.39:0/3463376769 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64b0107380 con 0x7f64b00fcc90 2023-06-22T17:26:32.948 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.946+0000 7f649effd700 1 -- 172.21.15.39:0/3463376769 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+21396 (secure 0 0 0) 0x7f64a4019090 con 0x7f64b00fcc90 2023-06-22T17:26:32.949 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-22T17:26:32.949 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":80,"fsid":"d88002e2-1120-11ee-9b2b-001a4aab830c","created":"2023-06-22T17:19:36.313754+0000","modified":"2023-06-22T17:26:32.174224+0000","last_up_change":"2023-06-22T17:25:51.023671+0000","last_in_change":"2023-06-22T17:25:33.400238+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-22T17:22:35.797853+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-06-22T17:26:30.187897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-06-22T17:26:31.538388+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"eb8f5cba-9266-47ec-8af1-8e9edd6ec96a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":453150307},{"type":"v1","addr":"172.21.15.39:6803","nonce":453150307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":453150307},{"type":"v1","addr":"172.21.15.39:6805","nonce":453150307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":453150307},{"type":"v1","addr":"172.21.15.39:6809","nonce":453150307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":453150307},{"type":"v1","addr":"172.21.15.39:6807","nonce":453150307}]},"public_addr":"172.21.15.39:6803/453150307","cluster_addr":"172.21.15.39:6805/453150307","heartbeat_back_addr":"172.21.15.39:6809/453150307","heartbeat_front_addr":"172.21.15.39:6807/453150307","state":["exists","up"]},{"osd":1,"uuid":"8b8dbe3e-fd72-4f92-98a2-e212df5366c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6800","nonce":2708845767},{"type":"v1","addr":"172.21.15.101:6801","nonce":2708845767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2708845767},{"type":"v1","addr":"172.21.15.101:6803","nonce":2708845767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2708845767},{"type":"v1","addr":"172.21.15.101:6807","nonce":2708845767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2708845767},{"type":"v1","addr":"172.21.15.101:6805","nonce":2708845767}]},"public_addr":"172.21.15.101:6801/2708845767","cluster_addr":"172.21.15.101:6803/2708845767","heartbeat_back_addr":"172.21.15.101:6807/2708845767","heartbeat_front_addr":"172.21.15.101:6805/2708845767","state":["exists","up"]},{"osd":2,"uuid":"7b25ade6-b5e2-4194-92c7-3baf8c4e9560","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":3632902853},{"type":"v1","addr":"172.21.15.133:6801","nonce":3632902853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":3632902853},{"type":"v1","addr":"172.21.15.133:6803","nonce":3632902853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":3632902853},{"type":"v1","addr":"172.21.15.133:6807","nonce":3632902853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":3632902853},{"type":"v1","addr":"172.21.15.133:6805","nonce":3632902853}]},"public_addr":"172.21.15.133:6801/3632902853","cluster_addr":"172.21.15.133:6803/3632902853","heartbeat_back_addr":"172.21.15.133:6807/3632902853","heartbeat_front_addr":"172.21.15.133:6805/3632902853","state":["exists","up"]},{"osd":3,"uuid":"b91b9111-cb8f-4c51-85c6-96c0851d57d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1215819990},{"type":"v1","addr":"172.21.15.39:6811","nonce":1215819990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1215819990},{"type":"v1","addr":"172.21.15.39:6813","nonce":1215819990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":1215819990},{"type":"v1","addr":"172.21.15.39:6817","nonce":1215819990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1215819990},{"type":"v1","addr":"172.21.15.39:6815","nonce":1215819990}]},"public_addr":"172.21.15.39:6811/1215819990","cluster_addr":"172.21.15.39:6813/1215819990","heartbeat_back_addr":"172.21.15.39:6817/1215819990","heartbeat_front_addr":"172.21.15.39:6815/1215819990","state":["exists","up"]},{"osd":4,"uuid":"14f6d038-1e3c-41e0-9be6-3615099ac5fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":1371317064},{"type":"v1","addr":"172.21.15.101:6809","nonce":1371317064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":1371317064},{"type":"v1","addr":"172.21.15.101:6811","nonce":1371317064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":1371317064},{"type":"v1","addr":"172.21.15.101:6815","nonce":1371317064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":1371317064},{"type":"v1","addr":"172.21.15.101:6813","nonce":1371317064}]},"public_addr":"172.21.15.101:6809/1371317064","cluster_addr":"172.21.15.101:6811/1371317064","heartbeat_back_addr":"172.21.15.101:6815/1371317064","heartbeat_front_addr":"172.21.15.101:6813/1371317064","state":["exists","up"]},{"osd":5,"uuid":"6daaeb92-c73d-4c30-bb1d-18d5305a63a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":747181981},{"type":"v1","addr":"172.21.15.133:6809","nonce":747181981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":747181981},{"type":"v1","addr":"172.21.15.133:6811","nonce":747181981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":747181981},{"type":"v1","addr":"172.21.15.133:6815","nonce":747181981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":747181981},{"type":"v1","addr":"172.21.15.133:6813","nonce":747181981}]},"public_addr":"172.21.15.133:6809/747181981","cluster_addr":"172.21.15.133:6811/747181981","heartbeat_back_addr":"172.21.15.133:6815/747181981","heartbeat_front_addr":"172.21.15.133:6813/747181981","state":["exists","up"]},{"osd":6,"uuid":"21f46569-8123-4bdd-b3c5-4cc8bc271899","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1221216882},{"type":"v1","addr":"172.21.15.39:6819","nonce":1221216882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1221216882},{"type":"v1","addr":"172.21.15.39:6821","nonce":1221216882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1221216882},{"type":"v1","addr":"172.21.15.39:6825","nonce":1221216882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1221216882},{"type":"v1","addr":"172.21.15.39:6823","nonce":1221216882}]},"public_addr":"172.21.15.39:6819/1221216882","cluster_addr":"172.21.15.39:6821/1221216882","heartbeat_back_addr":"172.21.15.39:6825/1221216882","heartbeat_front_addr":"172.21.15.39:6823/1221216882","state":["exists","up"]},{"osd":7,"uuid":"7a1cbc1b-8cf0-4153-b8e2-3d3313bed2ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":1606613660},{"type":"v1","addr":"172.21.15.101:6817","nonce":1606613660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":1606613660},{"type":"v1","addr":"172.21.15.101:6819","nonce":1606613660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":1606613660},{"type":"v1","addr":"172.21.15.101:6823","nonce":1606613660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":1606613660},{"type":"v1","addr":"172.21.15.101:6821","nonce":1606613660}]},"public_addr":"172.21.15.101:6817/1606613660","cluster_addr":"172.21.15.101:6819/1606613660","heartbeat_back_addr":"172.21.15.101:6823/1606613660","heartbeat_front_addr":"172.21.15.101:6821/1606613660","state":["exists","up"]},{"osd":8,"uuid":"ab09136e-4326-4c02-ac59-3d2264e85b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":1363381164},{"type":"v1","addr":"172.21.15.133:6817","nonce":1363381164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":1363381164},{"type":"v1","addr":"172.21.15.133:6819","nonce":1363381164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":1363381164},{"type":"v1","addr":"172.21.15.133:6823","nonce":1363381164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":1363381164},{"type":"v1","addr":"172.21.15.133:6821","nonce":1363381164}]},"public_addr":"172.21.15.133:6817/1363381164","cluster_addr":"172.21.15.133:6819/1363381164","heartbeat_back_addr":"172.21.15.133:6823/1363381164","heartbeat_front_addr":"172.21.15.133:6821/1363381164","state":["exists","up"]},{"osd":9,"uuid":"ec8c1344-0478-4094-b544-3360054fd92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":3010986736},{"type":"v1","addr":"172.21.15.39:6827","nonce":3010986736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3010986736},{"type":"v1","addr":"172.21.15.39:6829","nonce":3010986736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3010986736},{"type":"v1","addr":"172.21.15.39:6833","nonce":3010986736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":3010986736},{"type":"v1","addr":"172.21.15.39:6831","nonce":3010986736}]},"public_addr":"172.21.15.39:6827/3010986736","cluster_addr":"172.21.15.39:6829/3010986736","heartbeat_back_addr":"172.21.15.39:6833/3010986736","heartbeat_front_addr":"172.21.15.39:6831/3010986736","state":["exists","up"]},{"osd":10,"uuid":"9da2943f-b947-49d6-bab1-bc3a95299996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3982067769},{"type":"v1","addr":"172.21.15.101:6825","nonce":3982067769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":3982067769},{"type":"v1","addr":"172.21.15.101:6827","nonce":3982067769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":3982067769},{"type":"v1","addr":"172.21.15.101:6831","nonce":3982067769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":3982067769},{"type":"v1","addr":"172.21.15.101:6829","nonce":3982067769}]},"public_addr":"172.21.15.101:6825/3982067769","cluster_addr":"172.21.15.101:6827/3982067769","heartbeat_back_addr":"172.21.15.101:6831/3982067769","heartbeat_front_addr":"172.21.15.101:6829/3982067769","state":["exists","up"]},{"osd":11,"uuid":"ee5d51b4-c9ad-4603-a967-c60f3c1f2cdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":1265049047},{"type":"v1","addr":"172.21.15.133:6825","nonce":1265049047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6826","nonce":1265049047},{"type":"v1","addr":"172.21.15.133:6827","nonce":1265049047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6830","nonce":1265049047},{"type":"v1","addr":"172.21.15.133:6831","nonce":1265049047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6828","nonce":1265049047},{"type":"v1","addr":"172.21.15.133:6829","nonce":1265049047}]},"public_addr":"172.21.15.133:6825/1265049047","cluster_addr":"172.21.15.133:6827/1265049047","heartbeat_back_addr":"172.21.15.133:6831/1265049047","heartbeat_front_addr":"172.21.15.133:6829/1265049047","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:21:49.604087+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:22:11.298409+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:22:32.913915+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:22:56.095316+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:23:17.505593+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:23:38.490450+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:24:01.498630+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:24:22.952502+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:24:44.480122+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:25:07.141899+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:25:28.109970+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T17:25:49.227776+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:6801/1350590065":"2023-06-23T17:20:27.281966+0000","172.21.15.39:6800/1350590065":"2023-06-23T17:20:27.281966+0000","172.21.15.39:0/82302526":"2023-06-23T17:20:27.281966+0000","172.21.15.39:0/260892351":"2023-06-23T17:20:05.206927+0000","172.21.15.39:0/1513096987":"2023-06-23T17:20:05.206927+0000","172.21.15.39:6800/3960151496":"2023-06-23T17:20:05.206927+0000","172.21.15.39:6801/3960151496":"2023-06-23T17:20:05.206927+0000","172.21.15.39:0/2480502158":"2023-06-23T17:20:27.281966+0000","172.21.15.39:0/3848722689":"2023-06-23T17:20:27.281966+0000","172.21.15.39:0/2257210574":"2023-06-23T17:20:05.206927+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T17:26:32.951 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.949+0000 7f64af59e700 1 -- 172.21.15.39:0/3463376769 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f649809b680 msgr2=0x7f649809db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:32.951 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.949+0000 7f64af59e700 1 --2- 172.21.15.39:0/3463376769 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f649809b680 0x7f649809db30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f64a0005010 tx=0x7f64a0009e30 comp rx=0 tx=0).stop 2023-06-22T17:26:32.951 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.949+0000 7f64af59e700 1 -- 172.21.15.39:0/3463376769 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f64b00fcc90 msgr2=0x7f64b01125d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:32.951 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.949+0000 7f64af59e700 1 --2- 172.21.15.39:0/3463376769 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f64b00fcc90 0x7f64b01125d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f64a400ebc0 tx=0x7f64a400b040 comp rx=0 tx=0).stop 2023-06-22T17:26:32.951 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.949+0000 7f64af59e700 1 -- 172.21.15.39:0/3463376769 shutdown_connections 2023-06-22T17:26:32.952 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.949+0000 7f64af59e700 1 --2- 172.21.15.39:0/3463376769 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f649809b680 0x7f649809db30 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:32.952 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.949+0000 7f64af59e700 1 --2- 172.21.15.39:0/3463376769 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f64b00fd640 0x7f64b0114bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:32.952 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.949+0000 7f64af59e700 1 --2- 172.21.15.39:0/3463376769 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f64b00fb9e0 0x7f64b0065a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:32.952 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.949+0000 7f64af59e700 1 --2- 172.21.15.39:0/3463376769 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f64b00fcc90 0x7f64b01125d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:32.952 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.949+0000 7f64af59e700 1 -- 172.21.15.39:0/3463376769 >> 172.21.15.39:0/3463376769 conn(0x7f64b00f7140 msgr2=0x7f64b00f9230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:32.952 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.949+0000 7f64af59e700 1 -- 172.21.15.39:0/3463376769 shutdown_connections 2023-06-22T17:26:32.952 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:32.949+0000 7f64af59e700 1 -- 172.21.15.39:0/3463376769 wait complete. 2023-06-22T17:26:32.963 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-06-22T17:26:33.096 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.092+0000 7f38c81eb700 1 -- 172.21.15.39:0/845925064 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38c00fe970 msgr2=0x7f38c0105210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:33.096 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.092+0000 7f38c81eb700 1 --2- 172.21.15.39:0/845925064 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38c00fe970 0x7f38c0105210 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f38b8009d60 tx=0x7f38b8004d10 comp rx=0 tx=0).stop 2023-06-22T17:26:33.096 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.092+0000 7f38c81eb700 1 -- 172.21.15.39:0/845925064 shutdown_connections 2023-06-22T17:26:33.096 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.092+0000 7f38c81eb700 1 --2- 172.21.15.39:0/845925064 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38c00fe970 0x7f38c0105210 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:33.097 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.092+0000 7f38c81eb700 1 --2- 172.21.15.39:0/845925064 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f38c00fcd10 0x7f38c00fd120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:33.097 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.092+0000 7f38c81eb700 1 --2- 172.21.15.39:0/845925064 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f38c00fdfc0 0x7f38c00fe430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:33.097 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.092+0000 7f38c81eb700 1 -- 172.21.15.39:0/845925064 >> 172.21.15.39:0/845925064 conn(0x7f38c00f8490 msgr2=0x7f38c00fa8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:33.097 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.092+0000 7f38c81eb700 1 -- 172.21.15.39:0/845925064 shutdown_connections 2023-06-22T17:26:33.097 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.092+0000 7f38c81eb700 1 -- 172.21.15.39:0/845925064 wait complete. 2023-06-22T17:26:33.097 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.093+0000 7f38c81eb700 1 Processor -- start 2023-06-22T17:26:33.097 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.093+0000 7f38c81eb700 1 -- start start 2023-06-22T17:26:33.098 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.093+0000 7f38c81eb700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f38c00fcd10 0x7f38c010d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:33.098 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.093+0000 7f38c81eb700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38c00fdfc0 0x7f38c010dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:33.098 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.093+0000 7f38c81eb700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f38c00fe970 0x7f38c0114cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:33.098 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.093+0000 7f38c81eb700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c01152d0 con 0x7f38c00fdfc0 2023-06-22T17:26:33.098 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.093+0000 7f38c81eb700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c0115440 con 0x7f38c00fcd10 2023-06-22T17:26:33.098 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.093+0000 7f38c81eb700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c0115600 con 0x7f38c00fe970 2023-06-22T17:26:33.098 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.093+0000 7f38c5786700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38c00fdfc0 0x7f38c010dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:33.099 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.093+0000 7f38c5786700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38c00fdfc0 0x7f38c010dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37432/0 (socket says 172.21.15.39:37432) 2023-06-22T17:26:33.099 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.093+0000 7f38c5786700 1 -- 172.21.15.39:0/1532669643 learned_addr learned my addr 172.21.15.39:0/1532669643 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:26:33.099 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.093+0000 7f38c6788700 1 --2- 172.21.15.39:0/1532669643 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f38c00fe970 0x7f38c0114cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:33.099 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.093+0000 7f38c5f87700 1 --2- 172.21.15.39:0/1532669643 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f38c00fcd10 0x7f38c010d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:33.099 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.093+0000 7f38c5786700 1 -- 172.21.15.39:0/1532669643 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f38c00fe970 msgr2=0x7f38c0114cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:33.099 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.093+0000 7f38c5786700 1 --2- 172.21.15.39:0/1532669643 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f38c00fe970 0x7f38c0114cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:33.099 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.093+0000 7f38c5786700 1 -- 172.21.15.39:0/1532669643 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f38c00fcd10 msgr2=0x7f38c010d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:33.100 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.093+0000 7f38c5786700 1 --2- 172.21.15.39:0/1532669643 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f38c00fcd10 0x7f38c010d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:33.100 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.093+0000 7f38c5786700 1 -- 172.21.15.39:0/1532669643 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b8009940 con 0x7f38c00fdfc0 2023-06-22T17:26:33.100 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.093+0000 7f38c5786700 1 --2- 172.21.15.39:0/1532669643 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38c00fdfc0 0x7f38c010dc30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f38bc00ebc0 tx=0x7f38bc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:33.100 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.093+0000 7f38b27fc700 1 -- 172.21.15.39:0/1532669643 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38bc007cc0 con 0x7f38c00fdfc0 2023-06-22T17:26:33.100 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.094+0000 7f38b27fc700 1 -- 172.21.15.39:0/1532669643 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f38bc00f040 con 0x7f38c00fdfc0 2023-06-22T17:26:33.100 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.094+0000 7f38c81eb700 1 -- 172.21.15.39:0/1532669643 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38c0115880 con 0x7f38c00fdfc0 2023-06-22T17:26:33.100 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.094+0000 7f38b27fc700 1 -- 172.21.15.39:0/1532669643 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38bc00c930 con 0x7f38c00fdfc0 2023-06-22T17:26:33.101 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.094+0000 7f38c81eb700 1 -- 172.21.15.39:0/1532669643 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38c0115df0 con 0x7f38c00fdfc0 2023-06-22T17:26:33.101 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.095+0000 7f38c81eb700 1 -- 172.21.15.39:0/1532669643 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38c0103960 con 0x7f38c00fdfc0 2023-06-22T17:26:33.102 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.096+0000 7f38b27fc700 1 -- 172.21.15.39:0/1532669643 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f38bc0090d0 con 0x7f38c00fdfc0 2023-06-22T17:26:33.102 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.096+0000 7f38b27fc700 1 --2- 172.21.15.39:0/1532669643 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f38ac09b730 0x7f38ac09dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:33.102 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.096+0000 7f38c5f87700 1 --2- 172.21.15.39:0/1532669643 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f38ac09b730 0x7f38ac09dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:33.102 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.096+0000 7f38b27fc700 1 -- 172.21.15.39:0/1532669643 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f38bc0c49a0 con 0x7f38c00fdfc0 2023-06-22T17:26:33.102 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.097+0000 7f38c5f87700 1 --2- 172.21.15.39:0/1532669643 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f38ac09b730 0x7f38ac09dbe0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f38b4004f40 tx=0x7f38b400b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:33.103 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.100+0000 7f38b27fc700 1 -- 172.21.15.39:0/1532669643 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f38bc090350 con 0x7f38c00fdfc0 2023-06-22T17:26:33.351 INFO:teuthology.orchestra.run.smithi039.stderr:no valid command found; 1 closest matches: 2023-06-22T17:26:33.351 INFO:teuthology.orchestra.run.smithi039.stderr:fs set |refuse_client_session|max_xattr_size [--yes-i-really-mean-it] [--yes-i-really-really-mean-it] 2023-06-22T17:26:33.351 INFO:teuthology.orchestra.run.smithi039.stderr:Error EINVAL: invalid command 2023-06-22T17:26:33.354 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-06-22T17:26:33.355 DEBUG:tasks.cephfs.filesystem:fs_config: {'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2023-06-22T17:26:33.355 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2023-06-22T17:26:33.485 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.482+0000 7f607142e700 1 -- 172.21.15.39:0/2595200318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f606c0fbc20 msgr2=0x7f606c0fc030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:33.485 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.482+0000 7f607142e700 1 --2- 172.21.15.39:0/2595200318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f606c0fbc20 0x7f606c0fc030 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f605c0049c0 tx=0x7f605c009e30 comp rx=0 tx=0).stop 2023-06-22T17:26:33.485 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.482+0000 7f607142e700 1 -- 172.21.15.39:0/2595200318 shutdown_connections 2023-06-22T17:26:33.486 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.482+0000 7f607142e700 1 --2- 172.21.15.39:0/2595200318 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f606c0fd7c0 0x7f606c104080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:33.486 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.482+0000 7f607142e700 1 --2- 172.21.15.39:0/2595200318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f606c0fbc20 0x7f606c0fc030 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:33.486 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.482+0000 7f607142e700 1 --2- 172.21.15.39:0/2595200318 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f606c0fce30 0x7f606c0fd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:33.486 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.482+0000 7f607142e700 1 -- 172.21.15.39:0/2595200318 >> 172.21.15.39:0/2595200318 conn(0x7f606c0f7380 msgr2=0x7f606c0f97b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T17:26:33.486 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.482+0000 7f607142e700 1 -- 172.21.15.39:0/2595200318 shutdown_connections 2023-06-22T17:26:33.486 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.482+0000 7f607142e700 1 -- 172.21.15.39:0/2595200318 wait complete. 2023-06-22T17:26:33.488 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.483+0000 7f607142e700 1 Processor -- start 2023-06-22T17:26:33.488 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.483+0000 7f607142e700 1 -- start start 2023-06-22T17:26:33.488 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.483+0000 7f607142e700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f606c0fbc20 0x7f606c0666b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:33.488 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.483+0000 7f607142e700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f606c0fce30 0x7f606c114bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:33.488 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.483+0000 7f607142e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f606c0fd7c0 0x7f606c117100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:33.489 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.483+0000 7f607142e700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606c1177c0 con 0x7f606c0fd7c0 2023-06-22T17:26:33.489 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.483+0000 7f607142e700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606c117930 con 0x7f606c0fbc20 2023-06-22T17:26:33.489 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.484+0000 7f607142e700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606c117af0 con 0x7f606c0fce30 2023-06-22T17:26:33.489 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.484+0000 7f606b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f606c0fd7c0 0x7f606c117100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:33.489 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.484+0000 7f606affd700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f606c0fbc20 0x7f606c0666b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:33.489 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.484+0000 7f606a7fc700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f606c0fce30 0x7f606c114bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:33.489 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.484+0000 7f606affd700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f606c0fbc20 0x7f606c0666b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.39:55804/0 (socket says 172.21.15.39:55804) 2023-06-22T17:26:33.490 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.484+0000 7f606b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f606c0fd7c0 0x7f606c117100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37448/0 (socket says 172.21.15.39:37448) 2023-06-22T17:26:33.490 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.484+0000 7f606affd700 1 -- 172.21.15.39:0/1856646031 learned_addr learned my addr 172.21.15.39:0/1856646031 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-22T17:26:33.490 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.484+0000 7f606affd700 1 -- 172.21.15.39:0/1856646031 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f606c0fce30 msgr2=0x7f606c114bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:33.490 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.484+0000 7f606affd700 1 --2- 172.21.15.39:0/1856646031 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f606c0fce30 0x7f606c114bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:33.490 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.484+0000 7f606affd700 1 -- 172.21.15.39:0/1856646031 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f606c0fd7c0 msgr2=0x7f606c117100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T17:26:33.490 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.484+0000 7f606affd700 1 --2- 172.21.15.39:0/1856646031 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f606c0fd7c0 0x7f606c117100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T17:26:33.490 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.484+0000 7f606affd700 1 -- 172.21.15.39:0/1856646031 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f605c005040 con 0x7f606c0fbc20 2023-06-22T17:26:33.490 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.484+0000 7f606affd700 1 --2- 172.21.15.39:0/1856646031 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f606c0fbc20 0x7f606c0666b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f605c000cc0 tx=0x7f605c009e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:33.491 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.484+0000 7f60637fe700 1 -- 172.21.15.39:0/1856646031 <== mon.2 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605c00e040 con 0x7f606c0fbc20 2023-06-22T17:26:33.491 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.484+0000 7f60637fe700 1 -- 172.21.15.39:0/1856646031 <== mon.2 v2:172.21.15.101:3300/0 2 ==== config(27 keys) v1 ==== 1136+0+0 (secure 0 0 0) 0x7f605c009780 con 0x7f606c0fbc20 2023-06-22T17:26:33.491 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.484+0000 7f607142e700 1 -- 172.21.15.39:0/1856646031 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f606c117d10 con 0x7f606c0fbc20 2023-06-22T17:26:33.491 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.484+0000 7f60637fe700 1 -- 172.21.15.39:0/1856646031 <== mon.2 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605c013df0 con 0x7f606c0fbc20 2023-06-22T17:26:33.491 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.485+0000 7f607142e700 1 -- 172.21.15.39:0/1856646031 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f606c118170 con 0x7f606c0fbc20 2023-06-22T17:26:33.491 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.485+0000 7f607142e700 1 -- 172.21.15.39:0/1856646031 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f606c103000 con 0x7f606c0fbc20 2023-06-22T17:26:33.492 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.486+0000 7f60637fe700 1 -- 172.21.15.39:0/1856646031 <== mon.2 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f605c0135b0 con 0x7f606c0fbc20 2023-06-22T17:26:33.492 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.487+0000 7f60637fe700 1 --2- 172.21.15.39:0/1856646031 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f605809b730 0x7f605809dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T17:26:33.492 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.487+0000 7f60637fe700 1 -- 172.21.15.39:0/1856646031 <== mon.2 v2:172.21.15.101:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f605c0c5fe0 con 0x7f606c0fbc20 2023-06-22T17:26:33.492 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.487+0000 7f606a7fc700 1 --2- 172.21.15.39:0/1856646031 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f605809b730 0x7f605809dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T17:26:33.493 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.488+0000 7f606a7fc700 1 --2- 172.21.15.39:0/1856646031 >> [v2:172.21.15.39:6800/886078282,v1:172.21.15.39:6801/886078282] conn(0x7f605809b730 0x7f605809dbe0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6054004f40 tx=0x7f605400b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T17:26:33.493 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-22T17:26:33.490+0000 7f60637fe700 1 -- 172.21.15.39:0/1856646031 <== mon.2 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f605c091910 con 0x7f606c0fbc20 2023-06-22T17:26:33.683 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:33 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/3154074577' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-06-22T17:26:33.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:33 smithi039 ceph-mon[84805]: from='client.? 172.21.15.39:0/3463376769' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-06-22T17:26:33.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:33 smithi039 ceph-mon[84805]: mds.? [v2:172.21.15.133:6838/1006809426,v1:172.21.15.133:6839/1006809426] up:active 2023-06-22T17:26:33.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:33 smithi039 ceph-mon[84805]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-06-22T17:26:33.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:33 smithi039 ceph-mon[84805]: osdmap e81: 12 total, 12 up, 12 in 2023-06-22T17:26:33.684 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:33 smithi039 ceph-mon[84805]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-06-22T17:26:33.685 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:33 smithi039 ceph-mon[84805]: pgmap v254: 97 pgs: 6 creating+peering, 11 active+clean, 80 unknown; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:33.685 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:33 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:33.685 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:33 smithi039 ceph-mon[84805]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:33.744 INFO:teuthology.orchestra.run.smithi039.stderr:no valid command found; 1 closest matches: 2023-06-22T17:26:33.745 INFO:teuthology.orchestra.run.smithi039.stderr:fs set |refuse_client_session|max_xattr_size [--yes-i-really-mean-it] [--yes-i-really-really-mean-it] 2023-06-22T17:26:33.745 INFO:teuthology.orchestra.run.smithi039.stderr:Error EINVAL: invalid command 2023-06-22T17:26:33.748 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-06-22T17:26:33.749 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephadm.py", line 896, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 514, in __init__ self.create() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 717, in create self.set_allow_standby_replay(standby_replay) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 589, in set_allow_standby_replay self.set_var("allow_standby_replay", yes) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 574, in set_var self.mon_manager.raw_cluster_cmd("fs", "set", self.name, var, *a) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1624, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1615, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi039 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true' 2023-06-22T17:26:33.749 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-06-22T17:26:33.775 DEBUG:teuthology.orchestra.run.smithi101:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-06-22T17:26:33.799 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:33 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/3154074577' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-06-22T17:26:33.799 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:33 smithi101 ceph-mon[85507]: from='client.? 172.21.15.39:0/3463376769' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-06-22T17:26:33.800 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:33 smithi101 ceph-mon[85507]: mds.? [v2:172.21.15.133:6838/1006809426,v1:172.21.15.133:6839/1006809426] up:active 2023-06-22T17:26:33.800 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:33 smithi101 ceph-mon[85507]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-06-22T17:26:33.800 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:33 smithi101 ceph-mon[85507]: osdmap e81: 12 total, 12 up, 12 in 2023-06-22T17:26:33.800 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:33 smithi101 ceph-mon[85507]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-06-22T17:26:33.800 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:33 smithi101 ceph-mon[85507]: pgmap v254: 97 pgs: 6 creating+peering, 11 active+clean, 80 unknown; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:33.800 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:33 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:33.800 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:33 smithi101 ceph-mon[85507]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:33.803 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-06-22T17:26:33.827 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:33 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/3154074577' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-06-22T17:26:33.827 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:33 smithi133 ceph-mon[86268]: from='client.? 172.21.15.39:0/3463376769' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-06-22T17:26:33.827 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:33 smithi133 ceph-mon[86268]: mds.? [v2:172.21.15.133:6838/1006809426,v1:172.21.15.133:6839/1006809426] up:active 2023-06-22T17:26:33.827 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:33 smithi133 ceph-mon[86268]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-06-22T17:26:33.827 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:33 smithi133 ceph-mon[86268]: osdmap e81: 12 total, 12 up, 12 in 2023-06-22T17:26:33.827 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:33 smithi133 ceph-mon[86268]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2023-06-22T17:26:33.828 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:33 smithi133 ceph-mon[86268]: pgmap v254: 97 pgs: 6 creating+peering, 11 active+clean, 80 unknown; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-22T17:26:33.828 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:33 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:33.828 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:33 smithi133 ceph-mon[86268]: from='mgr.14150 172.21.15.39:0/4173196384' entity='mgr.x' 2023-06-22T17:26:33.831 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-06-22T17:26:33.831 DEBUG:teuthology.orchestra.run.smithi039:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-06-22T17:26:33.846 DEBUG:teuthology.orchestra.run.smithi101:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-06-22T17:26:33.861 DEBUG:teuthology.orchestra.run.smithi133:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-06-22T17:26:33.886 INFO:tasks.cephadm:Stopping all daemons... 2023-06-22T17:26:33.886 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2023-06-22T17:26:33.887 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mon.a 2023-06-22T17:26:34.183 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:33 smithi039 systemd[1]: Stopping Ceph mon.a for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:26:34.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:34 smithi039 conmon[84781]: 2023-06-22T17:26:34.244+0000 7f78296ea700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-06-22T17:26:34.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:34 smithi039 conmon[84781]: 2023-06-22T17:26:34.244+0000 7f78296ea700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2023-06-22T17:26:34.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Jun 22 17:26:34 smithi039 bash[132240]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c-mon-a 2023-06-22T17:26:34.902 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mon.a.service' 2023-06-22T17:26:34.946 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-22T17:26:34.946 INFO:tasks.cephadm.mon.a:Stopped mon.a 2023-06-22T17:26:34.947 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2023-06-22T17:26:34.947 DEBUG:teuthology.orchestra.run.smithi101:> sudo systemctl stop ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mon.b 2023-06-22T17:26:35.294 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:34 smithi101 systemd[1]: Stopping Ceph mon.b for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:26:35.294 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:35 smithi101 conmon[85482]: 2023-06-22T17:26:35.290+0000 7f996f73c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-06-22T17:26:35.295 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:35 smithi101 conmon[85482]: 2023-06-22T17:26:35.290+0000 7f996f73c700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2023-06-22T17:26:35.921 INFO:journalctl@ceph.mds.i.smithi133.stdout:Jun 22 17:26:35 smithi133 conmon[120573]: 2023-06-22T17:26:35.493+0000 7fd97ece5700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-06-22T17:26:36.126 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jun 22 17:26:35 smithi101 bash[120129]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c-mon-b 2023-06-22T17:26:36.299 DEBUG:teuthology.orchestra.run.smithi101:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mon.b.service' 2023-06-22T17:26:36.367 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-22T17:26:36.367 INFO:tasks.cephadm.mon.c:Stopped mon.b 2023-06-22T17:26:36.367 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2023-06-22T17:26:36.368 DEBUG:teuthology.orchestra.run.smithi133:> sudo systemctl stop ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mon.c 2023-06-22T17:26:36.671 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:36 smithi133 systemd[1]: Stopping Ceph mon.c for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:26:37.137 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:36 smithi133 conmon[86241]: 2023-06-22T17:26:36.792+0000 7ff1e4902700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-06-22T17:26:37.137 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:36 smithi133 conmon[86241]: 2023-06-22T17:26:36.792+0000 7ff1e4902700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2023-06-22T17:26:37.373 DEBUG:teuthology.orchestra.run.smithi133:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mon.c.service' 2023-06-22T17:26:37.397 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:37 smithi133 bash[122904]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c-mon-c 2023-06-22T17:26:37.397 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:37 smithi133 systemd[1]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mon.c.service: Succeeded. 2023-06-22T17:26:37.397 INFO:journalctl@ceph.mon.c.smithi133.stdout:Jun 22 17:26:37 smithi133 systemd[1]: Stopped Ceph mon.c for d88002e2-1120-11ee-9b2b-001a4aab830c. 2023-06-22T17:26:37.431 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-22T17:26:37.432 INFO:tasks.cephadm.mon.c:Stopped mon.c 2023-06-22T17:26:37.432 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2023-06-22T17:26:37.433 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mgr.x 2023-06-22T17:26:37.786 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:26:37 smithi039 systemd[1]: Stopping Ceph mgr.x for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:26:38.071 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Jun 22 17:26:37 smithi039 conmon[85217]: 2023-06-22T17:26:37.782+0000 7f033e3c1700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-22T17:26:38.282 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mgr.x.service' 2023-06-22T17:26:38.321 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-22T17:26:38.322 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2023-06-22T17:26:38.322 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2023-06-22T17:26:38.322 DEBUG:teuthology.orchestra.run.smithi101:> sudo systemctl stop ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mgr.y 2023-06-22T17:26:38.626 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jun 22 17:26:38 smithi101 systemd[1]: Stopping Ceph mgr.y for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:26:39.126 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jun 22 17:26:38 smithi101 bash[120908]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c-mgr-y 2023-06-22T17:26:39.126 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jun 22 17:26:39 smithi101 systemd[1]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mgr.y.service: Main process exited, code=exited, status=143/n/a 2023-06-22T17:26:39.161 DEBUG:teuthology.orchestra.run.smithi101:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mgr.y.service' 2023-06-22T17:26:39.197 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-22T17:26:39.197 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2023-06-22T17:26:39.198 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2023-06-22T17:26:39.198 DEBUG:teuthology.orchestra.run.smithi133:> sudo systemctl stop ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mgr.z 2023-06-22T17:26:39.643 INFO:journalctl@ceph.mgr.z.smithi133.stdout:Jun 22 17:26:39 smithi133 systemd[1]: Stopping Ceph mgr.z for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:26:39.921 INFO:journalctl@ceph.mgr.z.smithi133.stdout:Jun 22 17:26:39 smithi133 bash[123276]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c-mgr-z 2023-06-22T17:26:39.921 INFO:journalctl@ceph.mgr.z.smithi133.stdout:Jun 22 17:26:39 smithi133 systemd[1]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mgr.z.service: Main process exited, code=exited, status=143/n/a 2023-06-22T17:26:40.046 DEBUG:teuthology.orchestra.run.smithi133:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mgr.z.service' 2023-06-22T17:26:40.080 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-22T17:26:40.080 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2023-06-22T17:26:40.081 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2023-06-22T17:26:40.081 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.0 2023-06-22T17:26:40.433 INFO:journalctl@ceph.osd.0.smithi039.stdout:Jun 22 17:26:40 smithi039 systemd[1]: Stopping Ceph osd.0 for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:26:40.933 INFO:journalctl@ceph.osd.0.smithi039.stdout:Jun 22 17:26:40 smithi039 conmon[101634]: 2023-06-22T17:26:40.435+0000 7f4c2aff4700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-22T17:26:40.934 INFO:journalctl@ceph.osd.0.smithi039.stdout:Jun 22 17:26:40 smithi039 conmon[101634]: 2023-06-22T17:26:40.435+0000 7f4c2aff4700 -1 osd.0 81 *** Got signal Terminated *** 2023-06-22T17:26:40.934 INFO:journalctl@ceph.osd.0.smithi039.stdout:Jun 22 17:26:40 smithi039 conmon[101634]: 2023-06-22T17:26:40.435+0000 7f4c2aff4700 -1 osd.0 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-06-22T17:26:46.183 INFO:journalctl@ceph.osd.0.smithi039.stdout:Jun 22 17:26:46 smithi039 bash[133529]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c-osd-0 2023-06-22T17:26:47.458 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.0.service' 2023-06-22T17:26:47.497 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-22T17:26:47.497 INFO:tasks.cephadm.osd.0:Stopped osd.0 2023-06-22T17:26:47.498 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2023-06-22T17:26:47.498 DEBUG:teuthology.orchestra.run.smithi101:> sudo systemctl stop ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.1 2023-06-22T17:26:47.850 INFO:journalctl@ceph.osd.1.smithi101.stdout:Jun 22 17:26:47 smithi101 systemd[1]: Stopping Ceph osd.1 for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:26:48.127 INFO:journalctl@ceph.osd.1.smithi101.stdout:Jun 22 17:26:47 smithi101 conmon[91956]: 2023-06-22T17:26:47.847+0000 7f801c2b5700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-22T17:26:48.127 INFO:journalctl@ceph.osd.1.smithi101.stdout:Jun 22 17:26:47 smithi101 conmon[91956]: 2023-06-22T17:26:47.847+0000 7f801c2b5700 -1 osd.1 81 *** Got signal Terminated *** 2023-06-22T17:26:48.127 INFO:journalctl@ceph.osd.1.smithi101.stdout:Jun 22 17:26:47 smithi101 conmon[91956]: 2023-06-22T17:26:47.847+0000 7f801c2b5700 -1 osd.1 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-06-22T17:26:53.627 INFO:journalctl@ceph.osd.1.smithi101.stdout:Jun 22 17:26:53 smithi101 bash[121192]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c-osd-1 2023-06-22T17:26:54.979 DEBUG:teuthology.orchestra.run.smithi101:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.1.service' 2023-06-22T17:26:55.017 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-22T17:26:55.018 INFO:tasks.cephadm.osd.1:Stopped osd.1 2023-06-22T17:26:55.018 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2023-06-22T17:26:55.019 DEBUG:teuthology.orchestra.run.smithi133:> sudo systemctl stop ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.2 2023-06-22T17:26:55.375 INFO:journalctl@ceph.osd.2.smithi133.stdout:Jun 22 17:26:55 smithi133 systemd[1]: Stopping Ceph osd.2 for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:26:55.671 INFO:journalctl@ceph.osd.2.smithi133.stdout:Jun 22 17:26:55 smithi133 conmon[93974]: 2023-06-22T17:26:55.372+0000 7f318c6db700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-22T17:26:55.671 INFO:journalctl@ceph.osd.2.smithi133.stdout:Jun 22 17:26:55 smithi133 conmon[93974]: 2023-06-22T17:26:55.372+0000 7f318c6db700 -1 osd.2 81 *** Got signal Terminated *** 2023-06-22T17:26:55.671 INFO:journalctl@ceph.osd.2.smithi133.stdout:Jun 22 17:26:55 smithi133 conmon[93974]: 2023-06-22T17:26:55.372+0000 7f318c6db700 -1 osd.2 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-06-22T17:27:01.171 INFO:journalctl@ceph.osd.2.smithi133.stdout:Jun 22 17:27:01 smithi133 bash[123566]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c-osd-2 2023-06-22T17:27:02.580 DEBUG:teuthology.orchestra.run.smithi133:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.2.service' 2023-06-22T17:27:02.619 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-22T17:27:02.619 INFO:tasks.cephadm.osd.2:Stopped osd.2 2023-06-22T17:27:02.620 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2023-06-22T17:27:02.620 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.3 2023-06-22T17:27:02.933 INFO:journalctl@ceph.osd.3.smithi039.stdout:Jun 22 17:27:02 smithi039 systemd[1]: Stopping Ceph osd.3 for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:27:03.434 INFO:journalctl@ceph.osd.3.smithi039.stdout:Jun 22 17:27:02 smithi039 conmon[108578]: 2023-06-22T17:27:02.982+0000 7fdf0c564700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-22T17:27:03.434 INFO:journalctl@ceph.osd.3.smithi039.stdout:Jun 22 17:27:02 smithi039 conmon[108578]: 2023-06-22T17:27:02.982+0000 7fdf0c564700 -1 osd.3 81 *** Got signal Terminated *** 2023-06-22T17:27:03.434 INFO:journalctl@ceph.osd.3.smithi039.stdout:Jun 22 17:27:02 smithi039 conmon[108578]: 2023-06-22T17:27:02.982+0000 7fdf0c564700 -1 osd.3 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-06-22T17:27:06.684 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:06 smithi039 conmon[122618]: 2023-06-22T17:27:06.212+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:07.040 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:06 smithi133 conmon[115049]: 2023-06-22T17:27:06.825+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:07.041 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:27:06 smithi133 conmon[100972]: 2023-06-22T17:27:06.723+0000 7efe808b2700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:41.464305+0000 front 2023-06-22T17:26:41.464244+0000 (oldest deadline 2023-06-22T17:27:06.164481+0000) 2023-06-22T17:27:07.421 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:07 smithi133 conmon[107946]: 2023-06-22T17:27:07.037+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:07.433 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:07 smithi039 conmon[122618]: 2023-06-22T17:27:07.179+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:08.067 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:07 smithi133 conmon[115049]: 2023-06-22T17:27:07.832+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:08.068 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:27:07 smithi133 conmon[100972]: 2023-06-22T17:27:07.726+0000 7efe808b2700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:41.464305+0000 front 2023-06-22T17:26:41.464244+0000 (oldest deadline 2023-06-22T17:27:06.164481+0000) 2023-06-22T17:27:08.421 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:08 smithi133 conmon[107946]: 2023-06-22T17:27:08.065+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:08.434 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:08 smithi039 conmon[115622]: 2023-06-22T17:27:08.368+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.512440+0000 front 2023-06-22T17:26:43.512373+0000 (oldest deadline 2023-06-22T17:27:08.212584+0000) 2023-06-22T17:27:08.434 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:08 smithi039 conmon[122618]: 2023-06-22T17:27:08.156+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:08.933 INFO:journalctl@ceph.osd.3.smithi039.stdout:Jun 22 17:27:08 smithi039 bash[134078]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c-osd-3 2023-06-22T17:27:09.039 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:08 smithi133 conmon[115049]: 2023-06-22T17:27:08.862+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:09.040 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:27:08 smithi133 conmon[100972]: 2023-06-22T17:27:08.748+0000 7efe808b2700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:41.464305+0000 front 2023-06-22T17:26:41.464244+0000 (oldest deadline 2023-06-22T17:27:06.164481+0000) 2023-06-22T17:27:09.421 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:09 smithi133 conmon[107946]: 2023-06-22T17:27:09.038+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:09.434 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:09 smithi039 conmon[115622]: 2023-06-22T17:27:09.344+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.512440+0000 front 2023-06-22T17:26:43.512373+0000 (oldest deadline 2023-06-22T17:27:08.212584+0000) 2023-06-22T17:27:09.434 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:09 smithi039 conmon[122618]: 2023-06-22T17:27:09.176+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:10.083 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:09 smithi133 conmon[115049]: 2023-06-22T17:27:09.834+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:10.084 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:27:09 smithi133 conmon[100972]: 2023-06-22T17:27:09.710+0000 7efe808b2700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:41.464305+0000 front 2023-06-22T17:26:41.464244+0000 (oldest deadline 2023-06-22T17:27:06.164481+0000) 2023-06-22T17:27:10.099 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.3.service' 2023-06-22T17:27:10.137 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-22T17:27:10.138 INFO:tasks.cephadm.osd.3:Stopped osd.3 2023-06-22T17:27:10.138 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2023-06-22T17:27:10.139 DEBUG:teuthology.orchestra.run.smithi101:> sudo systemctl stop ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.4 2023-06-22T17:27:10.166 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:09 smithi101 conmon[112983]: 2023-06-22T17:27:09.894+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:10.421 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:10 smithi133 conmon[107946]: 2023-06-22T17:27:10.082+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:10.434 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:10 smithi039 conmon[115622]: 2023-06-22T17:27:10.317+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.512440+0000 front 2023-06-22T17:26:43.512373+0000 (oldest deadline 2023-06-22T17:27:08.212584+0000) 2023-06-22T17:27:10.434 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:10 smithi039 conmon[122618]: 2023-06-22T17:27:10.143+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:10.496 INFO:journalctl@ceph.osd.4.smithi101.stdout:Jun 22 17:27:10 smithi101 systemd[1]: Stopping Ceph osd.4 for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:27:10.875 INFO:journalctl@ceph.osd.4.smithi101.stdout:Jun 22 17:27:10 smithi101 conmon[98978]: 2023-06-22T17:27:10.494+0000 7f55a8f2c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-22T17:27:10.875 INFO:journalctl@ceph.osd.4.smithi101.stdout:Jun 22 17:27:10 smithi101 conmon[98978]: 2023-06-22T17:27:10.494+0000 7f55a8f2c700 -1 osd.4 81 *** Got signal Terminated *** 2023-06-22T17:27:10.875 INFO:journalctl@ceph.osd.4.smithi101.stdout:Jun 22 17:27:10 smithi101 conmon[98978]: 2023-06-22T17:27:10.494+0000 7f55a8f2c700 -1 osd.4 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-06-22T17:27:11.126 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:10 smithi101 conmon[112983]: 2023-06-22T17:27:10.873+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:11.127 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:10 smithi133 conmon[115049]: 2023-06-22T17:27:10.843+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:11.127 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:27:10 smithi133 conmon[100972]: 2023-06-22T17:27:10.716+0000 7efe808b2700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:41.464305+0000 front 2023-06-22T17:26:41.464244+0000 (oldest deadline 2023-06-22T17:27:06.164481+0000) 2023-06-22T17:27:11.421 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:11 smithi133 conmon[107946]: 2023-06-22T17:27:11.125+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:11.433 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:11 smithi039 conmon[122618]: 2023-06-22T17:27:11.109+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:11.434 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:11 smithi039 conmon[115622]: 2023-06-22T17:27:11.341+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.512440+0000 front 2023-06-22T17:26:43.512373+0000 (oldest deadline 2023-06-22T17:27:08.212584+0000) 2023-06-22T17:27:12.126 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:11 smithi101 conmon[112983]: 2023-06-22T17:27:11.866+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:12.171 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:27:11 smithi133 conmon[100972]: 2023-06-22T17:27:11.701+0000 7efe808b2700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:41.464305+0000 front 2023-06-22T17:26:41.464244+0000 (oldest deadline 2023-06-22T17:27:06.164481+0000) 2023-06-22T17:27:12.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:11 smithi133 conmon[115049]: 2023-06-22T17:27:11.815+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:12.433 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:12 smithi039 conmon[115622]: 2023-06-22T17:27:12.359+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.512440+0000 front 2023-06-22T17:26:43.512373+0000 (oldest deadline 2023-06-22T17:27:08.212584+0000) 2023-06-22T17:27:12.434 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:12 smithi039 conmon[122618]: 2023-06-22T17:27:12.109+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:12.666 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:12 smithi133 conmon[107946]: 2023-06-22T17:27:12.173+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:12.921 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:27:12 smithi133 conmon[100972]: 2023-06-22T17:27:12.664+0000 7efe808b2700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:41.464305+0000 front 2023-06-22T17:26:41.464244+0000 (oldest deadline 2023-06-22T17:27:06.164481+0000) 2023-06-22T17:27:12.921 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:12 smithi133 conmon[115049]: 2023-06-22T17:27:12.860+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:13.126 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:12 smithi101 conmon[112983]: 2023-06-22T17:27:12.833+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:13.410 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:13 smithi039 conmon[122618]: 2023-06-22T17:27:13.077+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:13.669 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:13 smithi133 conmon[107946]: 2023-06-22T17:27:13.171+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:13.683 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:13 smithi039 conmon[115622]: 2023-06-22T17:27:13.408+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.512440+0000 front 2023-06-22T17:26:43.512373+0000 (oldest deadline 2023-06-22T17:27:08.212584+0000) 2023-06-22T17:27:13.921 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:13 smithi133 conmon[115049]: 2023-06-22T17:27:13.816+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:13.921 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:27:13 smithi133 conmon[100972]: 2023-06-22T17:27:13.668+0000 7efe808b2700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:41.464305+0000 front 2023-06-22T17:26:41.464244+0000 (oldest deadline 2023-06-22T17:27:06.164481+0000) 2023-06-22T17:27:14.126 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:13 smithi101 conmon[112983]: 2023-06-22T17:27:13.803+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:14.418 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:14 smithi039 conmon[122618]: 2023-06-22T17:27:14.124+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:14.420 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:14 smithi133 conmon[107946]: 2023-06-22T17:27:14.155+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:14.683 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:14 smithi039 conmon[115622]: 2023-06-22T17:27:14.416+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.512440+0000 front 2023-06-22T17:26:43.512373+0000 (oldest deadline 2023-06-22T17:27:08.212584+0000) 2023-06-22T17:27:14.921 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:27:14 smithi133 conmon[100972]: 2023-06-22T17:27:14.631+0000 7efe808b2700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:41.464305+0000 front 2023-06-22T17:26:41.464244+0000 (oldest deadline 2023-06-22T17:27:06.164481+0000) 2023-06-22T17:27:14.921 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:14 smithi133 conmon[115049]: 2023-06-22T17:27:14.845+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:15.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:14 smithi101 conmon[112983]: 2023-06-22T17:27:14.820+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:15.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:14 smithi101 conmon[112983]: 2023-06-22T17:27:14.820+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:15.421 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:15 smithi133 conmon[107946]: 2023-06-22T17:27:15.152+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:15.433 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:15 smithi039 conmon[122618]: 2023-06-22T17:27:15.120+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:15.921 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:27:15 smithi133 conmon[100972]: 2023-06-22T17:27:15.633+0000 7efe808b2700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:41.464305+0000 front 2023-06-22T17:26:41.464244+0000 (oldest deadline 2023-06-22T17:27:06.164481+0000) 2023-06-22T17:27:15.921 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:15 smithi133 conmon[115049]: 2023-06-22T17:27:15.831+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:15.933 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:15 smithi039 conmon[115622]: 2023-06-22T17:27:15.452+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.512440+0000 front 2023-06-22T17:26:43.512373+0000 (oldest deadline 2023-06-22T17:27:08.212584+0000) 2023-06-22T17:27:16.127 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:15 smithi101 conmon[105890]: 2023-06-22T17:27:15.702+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.932693+0000 front 2023-06-22T17:26:49.932674+0000 (oldest deadline 2023-06-22T17:27:15.232896+0000) 2023-06-22T17:27:16.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:15 smithi101 conmon[112983]: 2023-06-22T17:27:15.850+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:16.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:15 smithi101 conmon[112983]: 2023-06-22T17:27:15.850+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:16.414 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:16 smithi039 conmon[122618]: 2023-06-22T17:27:16.090+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:16.415 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:16 smithi039 conmon[122618]: 2023-06-22T17:27:16.090+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:16.421 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:16 smithi133 conmon[107946]: 2023-06-22T17:27:16.132+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:16.626 INFO:journalctl@ceph.osd.4.smithi101.stdout:Jun 22 17:27:16 smithi101 bash[121740]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c-osd-4 2023-06-22T17:27:16.683 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:16 smithi039 conmon[115622]: 2023-06-22T17:27:16.412+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.512440+0000 front 2023-06-22T17:26:43.512373+0000 (oldest deadline 2023-06-22T17:27:08.212584+0000) 2023-06-22T17:27:16.921 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:16 smithi133 conmon[115049]: 2023-06-22T17:27:16.832+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:16.921 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:27:16 smithi133 conmon[100972]: 2023-06-22T17:27:16.611+0000 7efe808b2700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:41.464305+0000 front 2023-06-22T17:26:41.464244+0000 (oldest deadline 2023-06-22T17:27:06.164481+0000) 2023-06-22T17:27:16.921 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:27:16 smithi133 conmon[100972]: 2023-06-22T17:27:16.611+0000 7efe808b2700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.566629+0000 front 2023-06-22T17:26:52.566611+0000 (oldest deadline 2023-06-22T17:27:16.066581+0000) 2023-06-22T17:27:17.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:16 smithi101 conmon[112983]: 2023-06-22T17:27:16.846+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:17.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:16 smithi101 conmon[112983]: 2023-06-22T17:27:16.846+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:17.127 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:16 smithi101 conmon[105890]: 2023-06-22T17:27:16.687+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.932693+0000 front 2023-06-22T17:26:49.932674+0000 (oldest deadline 2023-06-22T17:27:15.232896+0000) 2023-06-22T17:27:17.421 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:17 smithi133 conmon[107946]: 2023-06-22T17:27:17.131+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:17.422 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:17 smithi039 conmon[122618]: 2023-06-22T17:27:17.074+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:17.422 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:17 smithi039 conmon[122618]: 2023-06-22T17:27:17.074+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:17.683 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:17 smithi039 conmon[115622]: 2023-06-22T17:27:17.418+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.512440+0000 front 2023-06-22T17:26:43.512373+0000 (oldest deadline 2023-06-22T17:27:08.212584+0000) 2023-06-22T17:27:17.697 DEBUG:teuthology.orchestra.run.smithi101:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.4.service' 2023-06-22T17:27:17.775 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-22T17:27:17.776 INFO:tasks.cephadm.osd.4:Stopped osd.4 2023-06-22T17:27:17.776 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2023-06-22T17:27:17.777 DEBUG:teuthology.orchestra.run.smithi133:> sudo systemctl stop ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.5 2023-06-22T17:27:17.921 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:17 smithi133 conmon[115049]: 2023-06-22T17:27:17.862+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:17.921 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:17 smithi133 conmon[115049]: 2023-06-22T17:27:17.862+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:17.921 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:27:17 smithi133 conmon[100972]: 2023-06-22T17:27:17.628+0000 7efe808b2700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:41.464305+0000 front 2023-06-22T17:26:41.464244+0000 (oldest deadline 2023-06-22T17:27:06.164481+0000) 2023-06-22T17:27:17.922 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:27:17 smithi133 conmon[100972]: 2023-06-22T17:27:17.628+0000 7efe808b2700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.566629+0000 front 2023-06-22T17:26:52.566611+0000 (oldest deadline 2023-06-22T17:27:16.066581+0000) 2023-06-22T17:27:17.922 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:27:17 smithi133 systemd[1]: Stopping Ceph osd.5 for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:27:18.127 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:17 smithi101 conmon[105890]: 2023-06-22T17:27:17.678+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.932693+0000 front 2023-06-22T17:26:49.932674+0000 (oldest deadline 2023-06-22T17:27:15.232896+0000) 2023-06-22T17:27:18.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:17 smithi101 conmon[112983]: 2023-06-22T17:27:17.829+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:18.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:17 smithi101 conmon[112983]: 2023-06-22T17:27:17.829+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:18.407 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:18 smithi039 conmon[122618]: 2023-06-22T17:27:18.041+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:18.407 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:18 smithi039 conmon[122618]: 2023-06-22T17:27:18.041+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:18.421 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:18 smithi133 conmon[107946]: 2023-06-22T17:27:18.110+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:18.421 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:18 smithi133 conmon[107946]: 2023-06-22T17:27:18.110+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.767866+0000 front 2023-06-22T17:26:52.767701+0000 (oldest deadline 2023-06-22T17:27:17.467693+0000) 2023-06-22T17:27:18.421 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:27:18 smithi133 conmon[100972]: 2023-06-22T17:27:18.127+0000 7efe862b8700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-22T17:27:18.421 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:27:18 smithi133 conmon[100972]: 2023-06-22T17:27:18.127+0000 7efe862b8700 -1 osd.5 81 *** Got signal Terminated *** 2023-06-22T17:27:18.422 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:27:18 smithi133 conmon[100972]: 2023-06-22T17:27:18.127+0000 7efe862b8700 -1 osd.5 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-06-22T17:27:18.684 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:18 smithi039 conmon[115622]: 2023-06-22T17:27:18.405+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.512440+0000 front 2023-06-22T17:26:43.512373+0000 (oldest deadline 2023-06-22T17:27:08.212584+0000) 2023-06-22T17:27:18.921 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:27:18 smithi133 conmon[100972]: 2023-06-22T17:27:18.621+0000 7efe808b2700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:41.464305+0000 front 2023-06-22T17:26:41.464244+0000 (oldest deadline 2023-06-22T17:27:06.164481+0000) 2023-06-22T17:27:18.921 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:27:18 smithi133 conmon[100972]: 2023-06-22T17:27:18.621+0000 7efe808b2700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.566629+0000 front 2023-06-22T17:26:52.566611+0000 (oldest deadline 2023-06-22T17:27:16.066581+0000) 2023-06-22T17:27:18.921 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:18 smithi133 conmon[115049]: 2023-06-22T17:27:18.866+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:18.922 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:18 smithi133 conmon[115049]: 2023-06-22T17:27:18.866+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:19.127 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:18 smithi101 conmon[105890]: 2023-06-22T17:27:18.661+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.932693+0000 front 2023-06-22T17:26:49.932674+0000 (oldest deadline 2023-06-22T17:27:15.232896+0000) 2023-06-22T17:27:19.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:18 smithi101 conmon[112983]: 2023-06-22T17:27:18.823+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:19.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:18 smithi101 conmon[112983]: 2023-06-22T17:27:18.823+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:19.391 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:19 smithi039 conmon[122618]: 2023-06-22T17:27:19.083+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:19.392 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:19 smithi039 conmon[122618]: 2023-06-22T17:27:19.083+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:19.421 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:19 smithi133 conmon[107946]: 2023-06-22T17:27:19.077+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:19.421 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:19 smithi133 conmon[107946]: 2023-06-22T17:27:19.077+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.767866+0000 front 2023-06-22T17:26:52.767701+0000 (oldest deadline 2023-06-22T17:27:17.467693+0000) 2023-06-22T17:27:19.683 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:19 smithi039 conmon[115622]: 2023-06-22T17:27:19.389+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.512440+0000 front 2023-06-22T17:26:43.512373+0000 (oldest deadline 2023-06-22T17:27:08.212584+0000) 2023-06-22T17:27:19.684 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:19 smithi039 conmon[115622]: 2023-06-22T17:27:19.389+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.015647+0000 front 2023-06-22T17:26:52.015497+0000 (oldest deadline 2023-06-22T17:27:19.016131+0000) 2023-06-22T17:27:19.921 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:27:19 smithi133 conmon[100972]: 2023-06-22T17:27:19.635+0000 7efe808b2700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:41.464305+0000 front 2023-06-22T17:26:41.464244+0000 (oldest deadline 2023-06-22T17:27:06.164481+0000) 2023-06-22T17:27:19.921 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:27:19 smithi133 conmon[100972]: 2023-06-22T17:27:19.635+0000 7efe808b2700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.566629+0000 front 2023-06-22T17:26:52.566611+0000 (oldest deadline 2023-06-22T17:27:16.066581+0000) 2023-06-22T17:27:19.922 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:19 smithi133 conmon[115049]: 2023-06-22T17:27:19.822+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:19.922 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:19 smithi133 conmon[115049]: 2023-06-22T17:27:19.822+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:20.127 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:19 smithi101 conmon[105890]: 2023-06-22T17:27:19.680+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.932693+0000 front 2023-06-22T17:26:49.932674+0000 (oldest deadline 2023-06-22T17:27:15.232896+0000) 2023-06-22T17:27:20.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:19 smithi101 conmon[112983]: 2023-06-22T17:27:19.868+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:20.128 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:19 smithi101 conmon[112983]: 2023-06-22T17:27:19.868+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:20.421 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:20 smithi133 conmon[107946]: 2023-06-22T17:27:20.108+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:20.421 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:20 smithi133 conmon[107946]: 2023-06-22T17:27:20.108+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.767866+0000 front 2023-06-22T17:26:52.767701+0000 (oldest deadline 2023-06-22T17:27:17.467693+0000) 2023-06-22T17:27:20.433 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:20 smithi039 conmon[122618]: 2023-06-22T17:27:20.049+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:20.434 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:20 smithi039 conmon[122618]: 2023-06-22T17:27:20.049+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:20.921 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:27:20 smithi133 conmon[100972]: 2023-06-22T17:27:20.662+0000 7efe808b2700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:41.464305+0000 front 2023-06-22T17:26:41.464244+0000 (oldest deadline 2023-06-22T17:27:06.164481+0000) 2023-06-22T17:27:20.921 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:27:20 smithi133 conmon[100972]: 2023-06-22T17:27:20.662+0000 7efe808b2700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.566629+0000 front 2023-06-22T17:26:52.566611+0000 (oldest deadline 2023-06-22T17:27:16.066581+0000) 2023-06-22T17:27:20.922 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:20 smithi133 conmon[115049]: 2023-06-22T17:27:20.810+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:20.922 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:20 smithi133 conmon[115049]: 2023-06-22T17:27:20.810+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:20.933 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:20 smithi039 conmon[115622]: 2023-06-22T17:27:20.435+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.512440+0000 front 2023-06-22T17:26:43.512373+0000 (oldest deadline 2023-06-22T17:27:08.212584+0000) 2023-06-22T17:27:20.934 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:20 smithi039 conmon[115622]: 2023-06-22T17:27:20.435+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.015647+0000 front 2023-06-22T17:26:52.015497+0000 (oldest deadline 2023-06-22T17:27:19.016131+0000) 2023-06-22T17:27:21.127 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:20 smithi101 conmon[105890]: 2023-06-22T17:27:20.727+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.932693+0000 front 2023-06-22T17:26:49.932674+0000 (oldest deadline 2023-06-22T17:27:15.232896+0000) 2023-06-22T17:27:21.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:20 smithi101 conmon[112983]: 2023-06-22T17:27:20.880+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:21.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:20 smithi101 conmon[112983]: 2023-06-22T17:27:20.880+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:21.398 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:21 smithi039 conmon[122618]: 2023-06-22T17:27:21.047+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:21.399 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:21 smithi039 conmon[122618]: 2023-06-22T17:27:21.047+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:21.421 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:21 smithi133 conmon[107946]: 2023-06-22T17:27:21.151+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:21.421 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:21 smithi133 conmon[107946]: 2023-06-22T17:27:21.151+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.767866+0000 front 2023-06-22T17:26:52.767701+0000 (oldest deadline 2023-06-22T17:27:17.467693+0000) 2023-06-22T17:27:21.683 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:21 smithi039 conmon[115622]: 2023-06-22T17:27:21.396+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.512440+0000 front 2023-06-22T17:26:43.512373+0000 (oldest deadline 2023-06-22T17:27:08.212584+0000) 2023-06-22T17:27:21.684 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:21 smithi039 conmon[115622]: 2023-06-22T17:27:21.396+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.015647+0000 front 2023-06-22T17:26:52.015497+0000 (oldest deadline 2023-06-22T17:27:19.016131+0000) 2023-06-22T17:27:22.114 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:21 smithi133 conmon[115049]: 2023-06-22T17:27:21.843+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:22.115 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:21 smithi133 conmon[115049]: 2023-06-22T17:27:21.843+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:22.115 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:27:21 smithi133 conmon[100972]: 2023-06-22T17:27:21.710+0000 7efe808b2700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:41.464305+0000 front 2023-06-22T17:26:41.464244+0000 (oldest deadline 2023-06-22T17:27:06.164481+0000) 2023-06-22T17:27:22.115 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:27:21 smithi133 conmon[100972]: 2023-06-22T17:27:21.710+0000 7efe808b2700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.566629+0000 front 2023-06-22T17:26:52.566611+0000 (oldest deadline 2023-06-22T17:27:16.066581+0000) 2023-06-22T17:27:22.126 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:21 smithi101 conmon[105890]: 2023-06-22T17:27:21.684+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.932693+0000 front 2023-06-22T17:26:49.932674+0000 (oldest deadline 2023-06-22T17:27:15.232896+0000) 2023-06-22T17:27:22.127 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:21 smithi101 conmon[105890]: 2023-06-22T17:27:21.684+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:55.233427+0000 front 2023-06-22T17:26:55.233506+0000 (oldest deadline 2023-06-22T17:27:21.133441+0000) 2023-06-22T17:27:22.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:21 smithi101 conmon[112983]: 2023-06-22T17:27:21.873+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:22.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:21 smithi101 conmon[112983]: 2023-06-22T17:27:21.873+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:22.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:21 smithi101 conmon[112983]: 2023-06-22T17:27:21.873+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:22.398 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:22 smithi039 conmon[122618]: 2023-06-22T17:27:22.049+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:22.398 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:22 smithi039 conmon[122618]: 2023-06-22T17:27:22.049+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:22.398 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:22 smithi039 conmon[122618]: 2023-06-22T17:27:22.049+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:27:00.343681+0000 front 2023-06-22T17:27:00.343658+0000 (oldest deadline 2023-06-22T17:27:21.443709+0000) 2023-06-22T17:27:22.421 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:22 smithi133 conmon[107946]: 2023-06-22T17:27:22.113+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:22.421 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:22 smithi133 conmon[107946]: 2023-06-22T17:27:22.113+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.767866+0000 front 2023-06-22T17:26:52.767701+0000 (oldest deadline 2023-06-22T17:27:17.467693+0000) 2023-06-22T17:27:22.421 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:22 smithi133 conmon[107946]: 2023-06-22T17:27:22.113+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:59.168703+0000 front 2023-06-22T17:26:59.168748+0000 (oldest deadline 2023-06-22T17:27:21.468879+0000) 2023-06-22T17:27:22.683 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:22 smithi039 conmon[115622]: 2023-06-22T17:27:22.396+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.512440+0000 front 2023-06-22T17:26:43.512373+0000 (oldest deadline 2023-06-22T17:27:08.212584+0000) 2023-06-22T17:27:22.684 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:22 smithi039 conmon[115622]: 2023-06-22T17:27:22.396+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.015647+0000 front 2023-06-22T17:26:52.015497+0000 (oldest deadline 2023-06-22T17:27:19.016131+0000) 2023-06-22T17:27:22.684 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:22 smithi039 conmon[115622]: 2023-06-22T17:27:22.396+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:59.016552+0000 front 2023-06-22T17:26:59.016657+0000 (oldest deadline 2023-06-22T17:27:21.916722+0000) 2023-06-22T17:27:23.127 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:22 smithi101 conmon[105890]: 2023-06-22T17:27:22.669+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.932693+0000 front 2023-06-22T17:26:49.932674+0000 (oldest deadline 2023-06-22T17:27:15.232896+0000) 2023-06-22T17:27:23.127 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:22 smithi101 conmon[105890]: 2023-06-22T17:27:22.669+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:55.233427+0000 front 2023-06-22T17:26:55.233506+0000 (oldest deadline 2023-06-22T17:27:21.133441+0000) 2023-06-22T17:27:23.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:22 smithi101 conmon[112983]: 2023-06-22T17:27:22.889+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:23.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:22 smithi101 conmon[112983]: 2023-06-22T17:27:22.889+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:23.128 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:22 smithi101 conmon[112983]: 2023-06-22T17:27:22.889+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:23.146 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:22 smithi133 conmon[115049]: 2023-06-22T17:27:22.840+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:23.147 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:22 smithi133 conmon[115049]: 2023-06-22T17:27:22.840+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:23.147 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:27:22 smithi133 conmon[100972]: 2023-06-22T17:27:22.679+0000 7efe808b2700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:41.464305+0000 front 2023-06-22T17:26:41.464244+0000 (oldest deadline 2023-06-22T17:27:06.164481+0000) 2023-06-22T17:27:23.147 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:27:22 smithi133 conmon[100972]: 2023-06-22T17:27:22.679+0000 7efe808b2700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.566629+0000 front 2023-06-22T17:26:52.566611+0000 (oldest deadline 2023-06-22T17:27:16.066581+0000) 2023-06-22T17:27:23.147 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:27:22 smithi133 conmon[100972]: 2023-06-22T17:27:22.679+0000 7efe808b2700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:56.567538+0000 front 2023-06-22T17:26:56.567493+0000 (oldest deadline 2023-06-22T17:27:22.467597+0000) 2023-06-22T17:27:23.421 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:23 smithi133 conmon[107946]: 2023-06-22T17:27:23.144+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:23.421 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:23 smithi133 conmon[107946]: 2023-06-22T17:27:23.144+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.767866+0000 front 2023-06-22T17:26:52.767701+0000 (oldest deadline 2023-06-22T17:27:17.467693+0000) 2023-06-22T17:27:23.421 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:23 smithi133 conmon[107946]: 2023-06-22T17:27:23.144+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:59.168703+0000 front 2023-06-22T17:26:59.168748+0000 (oldest deadline 2023-06-22T17:27:21.468879+0000) 2023-06-22T17:27:23.428 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:23 smithi039 conmon[122618]: 2023-06-22T17:27:23.072+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:23.428 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:23 smithi039 conmon[122618]: 2023-06-22T17:27:23.072+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:23.428 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:23 smithi039 conmon[122618]: 2023-06-22T17:27:23.072+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:27:00.343681+0000 front 2023-06-22T17:27:00.343658+0000 (oldest deadline 2023-06-22T17:27:21.443709+0000) 2023-06-22T17:27:23.683 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:23 smithi039 conmon[115622]: 2023-06-22T17:27:23.425+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.512440+0000 front 2023-06-22T17:26:43.512373+0000 (oldest deadline 2023-06-22T17:27:08.212584+0000) 2023-06-22T17:27:23.684 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:23 smithi039 conmon[115622]: 2023-06-22T17:27:23.425+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.015647+0000 front 2023-06-22T17:26:52.015497+0000 (oldest deadline 2023-06-22T17:27:19.016131+0000) 2023-06-22T17:27:23.684 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:23 smithi039 conmon[115622]: 2023-06-22T17:27:23.425+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:59.016552+0000 front 2023-06-22T17:26:59.016657+0000 (oldest deadline 2023-06-22T17:27:21.916722+0000) 2023-06-22T17:27:23.921 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:27:23 smithi133 bash[124107]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c-osd-5 2023-06-22T17:27:23.921 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:23 smithi133 conmon[115049]: 2023-06-22T17:27:23.885+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:23.921 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:23 smithi133 conmon[115049]: 2023-06-22T17:27:23.885+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:23.922 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:23 smithi133 conmon[115049]: 2023-06-22T17:27:23.885+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:24.126 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:23 smithi101 conmon[105890]: 2023-06-22T17:27:23.644+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.932693+0000 front 2023-06-22T17:26:49.932674+0000 (oldest deadline 2023-06-22T17:27:15.232896+0000) 2023-06-22T17:27:24.127 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:23 smithi101 conmon[105890]: 2023-06-22T17:27:23.644+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:55.233427+0000 front 2023-06-22T17:26:55.233506+0000 (oldest deadline 2023-06-22T17:27:21.133441+0000) 2023-06-22T17:27:24.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:23 smithi101 conmon[112983]: 2023-06-22T17:27:23.861+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:24.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:23 smithi101 conmon[112983]: 2023-06-22T17:27:23.861+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:24.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:23 smithi101 conmon[112983]: 2023-06-22T17:27:23.861+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:24.421 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:24 smithi133 conmon[107946]: 2023-06-22T17:27:24.122+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:24.422 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:24 smithi133 conmon[107946]: 2023-06-22T17:27:24.122+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.767866+0000 front 2023-06-22T17:26:52.767701+0000 (oldest deadline 2023-06-22T17:27:17.467693+0000) 2023-06-22T17:27:24.422 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:24 smithi133 conmon[107946]: 2023-06-22T17:27:24.122+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:59.168703+0000 front 2023-06-22T17:26:59.168748+0000 (oldest deadline 2023-06-22T17:27:21.468879+0000) 2023-06-22T17:27:24.433 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:24 smithi039 conmon[122618]: 2023-06-22T17:27:24.055+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:24.434 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:24 smithi039 conmon[122618]: 2023-06-22T17:27:24.055+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:24.434 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:24 smithi039 conmon[122618]: 2023-06-22T17:27:24.055+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:27:00.343681+0000 front 2023-06-22T17:27:00.343658+0000 (oldest deadline 2023-06-22T17:27:21.443709+0000) 2023-06-22T17:27:24.934 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:24 smithi039 conmon[115622]: 2023-06-22T17:27:24.464+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.512440+0000 front 2023-06-22T17:26:43.512373+0000 (oldest deadline 2023-06-22T17:27:08.212584+0000) 2023-06-22T17:27:24.934 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:24 smithi039 conmon[115622]: 2023-06-22T17:27:24.464+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.015647+0000 front 2023-06-22T17:26:52.015497+0000 (oldest deadline 2023-06-22T17:27:19.016131+0000) 2023-06-22T17:27:24.934 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:24 smithi039 conmon[115622]: 2023-06-22T17:27:24.464+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:59.016552+0000 front 2023-06-22T17:26:59.016657+0000 (oldest deadline 2023-06-22T17:27:21.916722+0000) 2023-06-22T17:27:25.085 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:24 smithi133 conmon[115049]: 2023-06-22T17:27:24.919+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:25.085 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:24 smithi133 conmon[115049]: 2023-06-22T17:27:24.919+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:25.085 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:24 smithi133 conmon[115049]: 2023-06-22T17:27:24.919+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:25.127 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:24 smithi101 conmon[105890]: 2023-06-22T17:27:24.649+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.932693+0000 front 2023-06-22T17:26:49.932674+0000 (oldest deadline 2023-06-22T17:27:15.232896+0000) 2023-06-22T17:27:25.127 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:24 smithi101 conmon[105890]: 2023-06-22T17:27:24.649+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:55.233427+0000 front 2023-06-22T17:26:55.233506+0000 (oldest deadline 2023-06-22T17:27:21.133441+0000) 2023-06-22T17:27:25.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:24 smithi101 conmon[112983]: 2023-06-22T17:27:24.827+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:25.128 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:24 smithi101 conmon[112983]: 2023-06-22T17:27:24.827+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:25.128 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:24 smithi101 conmon[112983]: 2023-06-22T17:27:24.827+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:25.336 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:27:25 smithi133 systemd[1]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.5.service: Succeeded. 2023-06-22T17:27:25.336 INFO:journalctl@ceph.osd.5.smithi133.stdout:Jun 22 17:27:25 smithi133 systemd[1]: Stopped Ceph osd.5 for d88002e2-1120-11ee-9b2b-001a4aab830c. 2023-06-22T17:27:25.337 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:25 smithi133 conmon[107946]: 2023-06-22T17:27:25.099+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:25.337 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:25 smithi133 conmon[107946]: 2023-06-22T17:27:25.099+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.767866+0000 front 2023-06-22T17:26:52.767701+0000 (oldest deadline 2023-06-22T17:27:17.467693+0000) 2023-06-22T17:27:25.337 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:25 smithi133 conmon[107946]: 2023-06-22T17:27:25.099+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:59.168703+0000 front 2023-06-22T17:26:59.168748+0000 (oldest deadline 2023-06-22T17:27:21.468879+0000) 2023-06-22T17:27:25.338 DEBUG:teuthology.orchestra.run.smithi133:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.5.service' 2023-06-22T17:27:25.376 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-22T17:27:25.377 INFO:tasks.cephadm.osd.5:Stopped osd.5 2023-06-22T17:27:25.377 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2023-06-22T17:27:25.378 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.6 2023-06-22T17:27:25.407 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:25 smithi039 conmon[122618]: 2023-06-22T17:27:25.061+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:25.407 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:25 smithi039 conmon[122618]: 2023-06-22T17:27:25.061+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:25.407 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:25 smithi039 conmon[122618]: 2023-06-22T17:27:25.061+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:27:00.343681+0000 front 2023-06-22T17:27:00.343658+0000 (oldest deadline 2023-06-22T17:27:21.443709+0000) 2023-06-22T17:27:25.684 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:25 smithi039 systemd[1]: Stopping Ceph osd.6 for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:27:25.684 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:25 smithi039 conmon[115622]: 2023-06-22T17:27:25.512+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.512440+0000 front 2023-06-22T17:26:43.512373+0000 (oldest deadline 2023-06-22T17:27:08.212584+0000) 2023-06-22T17:27:25.684 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:25 smithi039 conmon[115622]: 2023-06-22T17:27:25.512+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.015647+0000 front 2023-06-22T17:26:52.015497+0000 (oldest deadline 2023-06-22T17:27:19.016131+0000) 2023-06-22T17:27:25.684 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:25 smithi039 conmon[115622]: 2023-06-22T17:27:25.512+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:59.016552+0000 front 2023-06-22T17:26:59.016657+0000 (oldest deadline 2023-06-22T17:27:21.916722+0000) 2023-06-22T17:27:25.877 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:25 smithi101 conmon[105890]: 2023-06-22T17:27:25.619+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.932693+0000 front 2023-06-22T17:26:49.932674+0000 (oldest deadline 2023-06-22T17:27:15.232896+0000) 2023-06-22T17:27:25.877 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:25 smithi101 conmon[105890]: 2023-06-22T17:27:25.619+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:55.233427+0000 front 2023-06-22T17:26:55.233506+0000 (oldest deadline 2023-06-22T17:27:21.133441+0000) 2023-06-22T17:27:25.877 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:25 smithi101 conmon[112983]: 2023-06-22T17:27:25.793+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:25.877 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:25 smithi101 conmon[112983]: 2023-06-22T17:27:25.793+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:25.878 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:25 smithi101 conmon[112983]: 2023-06-22T17:27:25.793+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:26.183 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:25 smithi039 conmon[115622]: 2023-06-22T17:27:25.862+0000 7fb4c3a98700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-22T17:27:26.184 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:25 smithi039 conmon[115622]: 2023-06-22T17:27:25.862+0000 7fb4c3a98700 -1 osd.6 81 *** Got signal Terminated *** 2023-06-22T17:27:26.184 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:25 smithi039 conmon[115622]: 2023-06-22T17:27:25.862+0000 7fb4c3a98700 -1 osd.6 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-06-22T17:27:26.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:26 smithi039 conmon[122618]: 2023-06-22T17:27:26.034+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:26.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:26 smithi039 conmon[122618]: 2023-06-22T17:27:26.034+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:26.185 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:26 smithi039 conmon[122618]: 2023-06-22T17:27:26.034+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:27:00.343681+0000 front 2023-06-22T17:27:00.343658+0000 (oldest deadline 2023-06-22T17:27:21.443709+0000) 2023-06-22T17:27:26.421 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:26 smithi133 conmon[107946]: 2023-06-22T17:27:26.104+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:26.421 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:26 smithi133 conmon[107946]: 2023-06-22T17:27:26.104+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.767866+0000 front 2023-06-22T17:26:52.767701+0000 (oldest deadline 2023-06-22T17:27:17.467693+0000) 2023-06-22T17:27:26.421 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:26 smithi133 conmon[107946]: 2023-06-22T17:27:26.104+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:59.168703+0000 front 2023-06-22T17:26:59.168748+0000 (oldest deadline 2023-06-22T17:27:21.468879+0000) 2023-06-22T17:27:26.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:25 smithi133 conmon[115049]: 2023-06-22T17:27:25.965+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:26.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:25 smithi133 conmon[115049]: 2023-06-22T17:27:25.965+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:26.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:25 smithi133 conmon[115049]: 2023-06-22T17:27:25.965+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:26.934 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:26 smithi039 conmon[115622]: 2023-06-22T17:27:26.478+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.512440+0000 front 2023-06-22T17:26:43.512373+0000 (oldest deadline 2023-06-22T17:27:08.212584+0000) 2023-06-22T17:27:26.934 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:26 smithi039 conmon[115622]: 2023-06-22T17:27:26.478+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.015647+0000 front 2023-06-22T17:26:52.015497+0000 (oldest deadline 2023-06-22T17:27:19.016131+0000) 2023-06-22T17:27:26.934 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:26 smithi039 conmon[115622]: 2023-06-22T17:27:26.478+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:59.016552+0000 front 2023-06-22T17:26:59.016657+0000 (oldest deadline 2023-06-22T17:27:21.916722+0000) 2023-06-22T17:27:27.127 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:26 smithi101 conmon[105890]: 2023-06-22T17:27:26.641+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.932693+0000 front 2023-06-22T17:26:49.932674+0000 (oldest deadline 2023-06-22T17:27:15.232896+0000) 2023-06-22T17:27:27.127 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:26 smithi101 conmon[105890]: 2023-06-22T17:27:26.641+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:55.233427+0000 front 2023-06-22T17:26:55.233506+0000 (oldest deadline 2023-06-22T17:27:21.133441+0000) 2023-06-22T17:27:27.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:26 smithi101 conmon[112983]: 2023-06-22T17:27:26.776+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:27.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:26 smithi101 conmon[112983]: 2023-06-22T17:27:26.776+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:27.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:26 smithi101 conmon[112983]: 2023-06-22T17:27:26.776+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:27.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:26 smithi133 conmon[115049]: 2023-06-22T17:27:26.968+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:27.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:26 smithi133 conmon[115049]: 2023-06-22T17:27:26.968+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:27.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:26 smithi133 conmon[115049]: 2023-06-22T17:27:26.968+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:27.422 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:27 smithi133 conmon[107946]: 2023-06-22T17:27:27.065+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:27.422 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:27 smithi133 conmon[107946]: 2023-06-22T17:27:27.065+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.767866+0000 front 2023-06-22T17:26:52.767701+0000 (oldest deadline 2023-06-22T17:27:17.467693+0000) 2023-06-22T17:27:27.422 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:27 smithi133 conmon[107946]: 2023-06-22T17:27:27.065+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:59.168703+0000 front 2023-06-22T17:26:59.168748+0000 (oldest deadline 2023-06-22T17:27:21.468879+0000) 2023-06-22T17:27:27.433 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:26 smithi039 conmon[122618]: 2023-06-22T17:27:26.992+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:27.433 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:26 smithi039 conmon[122618]: 2023-06-22T17:27:26.993+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:27.434 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:26 smithi039 conmon[122618]: 2023-06-22T17:27:26.993+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:27:00.343681+0000 front 2023-06-22T17:27:00.343658+0000 (oldest deadline 2023-06-22T17:27:21.443709+0000) 2023-06-22T17:27:27.933 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:27 smithi039 conmon[115622]: 2023-06-22T17:27:27.438+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.512440+0000 front 2023-06-22T17:26:43.512373+0000 (oldest deadline 2023-06-22T17:27:08.212584+0000) 2023-06-22T17:27:27.934 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:27 smithi039 conmon[115622]: 2023-06-22T17:27:27.438+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.015647+0000 front 2023-06-22T17:26:52.015497+0000 (oldest deadline 2023-06-22T17:27:19.016131+0000) 2023-06-22T17:27:27.934 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:27 smithi039 conmon[115622]: 2023-06-22T17:27:27.438+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:59.016552+0000 front 2023-06-22T17:26:59.016657+0000 (oldest deadline 2023-06-22T17:27:21.916722+0000) 2023-06-22T17:27:28.126 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:27 smithi101 conmon[105890]: 2023-06-22T17:27:27.686+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.932693+0000 front 2023-06-22T17:26:49.932674+0000 (oldest deadline 2023-06-22T17:27:15.232896+0000) 2023-06-22T17:27:28.127 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:27 smithi101 conmon[105890]: 2023-06-22T17:27:27.686+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:55.233427+0000 front 2023-06-22T17:26:55.233506+0000 (oldest deadline 2023-06-22T17:27:21.133441+0000) 2023-06-22T17:27:28.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:27 smithi101 conmon[112983]: 2023-06-22T17:27:27.745+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:28.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:27 smithi101 conmon[112983]: 2023-06-22T17:27:27.745+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:28.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:27 smithi101 conmon[112983]: 2023-06-22T17:27:27.745+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:28.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:27 smithi133 conmon[115049]: 2023-06-22T17:27:27.922+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:28.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:27 smithi133 conmon[115049]: 2023-06-22T17:27:27.922+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:28.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:27 smithi133 conmon[115049]: 2023-06-22T17:27:27.922+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:28.422 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:28 smithi133 conmon[107946]: 2023-06-22T17:27:28.018+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:28.422 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:28 smithi133 conmon[107946]: 2023-06-22T17:27:28.018+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.767866+0000 front 2023-06-22T17:26:52.767701+0000 (oldest deadline 2023-06-22T17:27:17.467693+0000) 2023-06-22T17:27:28.422 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:28 smithi133 conmon[107946]: 2023-06-22T17:27:28.018+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:59.168703+0000 front 2023-06-22T17:26:59.168748+0000 (oldest deadline 2023-06-22T17:27:21.468879+0000) 2023-06-22T17:27:28.433 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:27 smithi039 conmon[122618]: 2023-06-22T17:27:27.944+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:28.434 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:27 smithi039 conmon[122618]: 2023-06-22T17:27:27.944+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:28.434 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:27 smithi039 conmon[122618]: 2023-06-22T17:27:27.944+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:27:00.343681+0000 front 2023-06-22T17:27:00.343658+0000 (oldest deadline 2023-06-22T17:27:21.443709+0000) 2023-06-22T17:27:28.918 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:28 smithi039 conmon[115622]: 2023-06-22T17:27:28.485+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.512440+0000 front 2023-06-22T17:26:43.512373+0000 (oldest deadline 2023-06-22T17:27:08.212584+0000) 2023-06-22T17:27:28.918 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:28 smithi039 conmon[115622]: 2023-06-22T17:27:28.485+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.015647+0000 front 2023-06-22T17:26:52.015497+0000 (oldest deadline 2023-06-22T17:27:19.016131+0000) 2023-06-22T17:27:28.918 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:28 smithi039 conmon[115622]: 2023-06-22T17:27:28.485+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:59.016552+0000 front 2023-06-22T17:26:59.016657+0000 (oldest deadline 2023-06-22T17:27:21.916722+0000) 2023-06-22T17:27:29.127 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:28 smithi101 conmon[105890]: 2023-06-22T17:27:28.640+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.932693+0000 front 2023-06-22T17:26:49.932674+0000 (oldest deadline 2023-06-22T17:27:15.232896+0000) 2023-06-22T17:27:29.127 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:28 smithi101 conmon[105890]: 2023-06-22T17:27:28.640+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:55.233427+0000 front 2023-06-22T17:26:55.233506+0000 (oldest deadline 2023-06-22T17:27:21.133441+0000) 2023-06-22T17:27:29.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:28 smithi101 conmon[112983]: 2023-06-22T17:27:28.715+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:29.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:28 smithi101 conmon[112983]: 2023-06-22T17:27:28.715+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:29.128 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:28 smithi101 conmon[112983]: 2023-06-22T17:27:28.715+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:29.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:28 smithi133 conmon[115049]: 2023-06-22T17:27:28.900+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:29.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:28 smithi133 conmon[115049]: 2023-06-22T17:27:28.900+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:29.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:28 smithi133 conmon[115049]: 2023-06-22T17:27:28.900+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:29.171 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:29 smithi133 conmon[107946]: 2023-06-22T17:27:29.001+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:29.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:29 smithi133 conmon[107946]: 2023-06-22T17:27:29.001+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.767866+0000 front 2023-06-22T17:26:52.767701+0000 (oldest deadline 2023-06-22T17:27:17.467693+0000) 2023-06-22T17:27:29.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:29 smithi133 conmon[107946]: 2023-06-22T17:27:29.001+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:59.168703+0000 front 2023-06-22T17:26:59.168748+0000 (oldest deadline 2023-06-22T17:27:21.468879+0000) 2023-06-22T17:27:29.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:28 smithi039 conmon[122618]: 2023-06-22T17:27:28.915+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:29.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:28 smithi039 conmon[122618]: 2023-06-22T17:27:28.915+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:29.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:28 smithi039 conmon[122618]: 2023-06-22T17:27:28.915+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:27:00.343681+0000 front 2023-06-22T17:27:00.343658+0000 (oldest deadline 2023-06-22T17:27:21.443709+0000) 2023-06-22T17:27:29.934 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:29 smithi039 conmon[115622]: 2023-06-22T17:27:29.477+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.512440+0000 front 2023-06-22T17:26:43.512373+0000 (oldest deadline 2023-06-22T17:27:08.212584+0000) 2023-06-22T17:27:29.934 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:29 smithi039 conmon[115622]: 2023-06-22T17:27:29.477+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.015647+0000 front 2023-06-22T17:26:52.015497+0000 (oldest deadline 2023-06-22T17:27:19.016131+0000) 2023-06-22T17:27:29.934 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:29 smithi039 conmon[115622]: 2023-06-22T17:27:29.477+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:59.016552+0000 front 2023-06-22T17:26:59.016657+0000 (oldest deadline 2023-06-22T17:27:21.916722+0000) 2023-06-22T17:27:29.935 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:29 smithi039 conmon[115622]: 2023-06-22T17:27:29.477+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:05.318294+0000 front 2023-06-22T17:27:05.318270+0000 (oldest deadline 2023-06-22T17:27:29.418459+0000) 2023-06-22T17:27:30.127 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:29 smithi101 conmon[105890]: 2023-06-22T17:27:29.632+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.932693+0000 front 2023-06-22T17:26:49.932674+0000 (oldest deadline 2023-06-22T17:27:15.232896+0000) 2023-06-22T17:27:30.127 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:29 smithi101 conmon[105890]: 2023-06-22T17:27:29.632+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:55.233427+0000 front 2023-06-22T17:26:55.233506+0000 (oldest deadline 2023-06-22T17:27:21.133441+0000) 2023-06-22T17:27:30.127 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:29 smithi101 conmon[105890]: 2023-06-22T17:27:29.632+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:06.434910+0000 front 2023-06-22T17:27:06.434807+0000 (oldest deadline 2023-06-22T17:27:28.734765+0000) 2023-06-22T17:27:30.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:29 smithi101 conmon[112983]: 2023-06-22T17:27:29.749+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:30.128 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:29 smithi101 conmon[112983]: 2023-06-22T17:27:29.749+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:30.128 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:29 smithi101 conmon[112983]: 2023-06-22T17:27:29.749+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:30.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:29 smithi133 conmon[115049]: 2023-06-22T17:27:29.915+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:30.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:29 smithi133 conmon[115049]: 2023-06-22T17:27:29.915+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:30.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:29 smithi133 conmon[115049]: 2023-06-22T17:27:29.915+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:30.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:29 smithi133 conmon[107946]: 2023-06-22T17:27:29.993+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:30.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:29 smithi133 conmon[107946]: 2023-06-22T17:27:29.993+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.767866+0000 front 2023-06-22T17:26:52.767701+0000 (oldest deadline 2023-06-22T17:27:17.467693+0000) 2023-06-22T17:27:30.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:29 smithi133 conmon[107946]: 2023-06-22T17:27:29.993+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:59.168703+0000 front 2023-06-22T17:26:59.168748+0000 (oldest deadline 2023-06-22T17:27:21.468879+0000) 2023-06-22T17:27:30.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:29 smithi133 conmon[107946]: 2023-06-22T17:27:29.993+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.969905+0000 front 2023-06-22T17:27:04.970073+0000 (oldest deadline 2023-06-22T17:27:29.669924+0000) 2023-06-22T17:27:30.434 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:29 smithi039 conmon[122618]: 2023-06-22T17:27:29.953+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:30.434 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:29 smithi039 conmon[122618]: 2023-06-22T17:27:29.953+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:30.434 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:29 smithi039 conmon[122618]: 2023-06-22T17:27:29.953+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:27:00.343681+0000 front 2023-06-22T17:27:00.343658+0000 (oldest deadline 2023-06-22T17:27:21.443709+0000) 2023-06-22T17:27:30.877 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:30 smithi101 conmon[105890]: 2023-06-22T17:27:30.584+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.932693+0000 front 2023-06-22T17:26:49.932674+0000 (oldest deadline 2023-06-22T17:27:15.232896+0000) 2023-06-22T17:27:30.877 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:30 smithi101 conmon[105890]: 2023-06-22T17:27:30.584+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:55.233427+0000 front 2023-06-22T17:26:55.233506+0000 (oldest deadline 2023-06-22T17:27:21.133441+0000) 2023-06-22T17:27:30.877 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:30 smithi101 conmon[105890]: 2023-06-22T17:27:30.584+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:06.434910+0000 front 2023-06-22T17:27:06.434807+0000 (oldest deadline 2023-06-22T17:27:28.734765+0000) 2023-06-22T17:27:30.877 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:30 smithi101 conmon[112983]: 2023-06-22T17:27:30.797+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:30.878 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:30 smithi101 conmon[112983]: 2023-06-22T17:27:30.797+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:30.878 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:30 smithi101 conmon[112983]: 2023-06-22T17:27:30.797+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:30.878 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:30 smithi101 conmon[112983]: 2023-06-22T17:27:30.797+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.163707+0000 front 2023-06-22T17:27:04.163629+0000 (oldest deadline 2023-06-22T17:27:30.564301+0000) 2023-06-22T17:27:30.918 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:30 smithi039 conmon[115622]: 2023-06-22T17:27:30.457+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.512440+0000 front 2023-06-22T17:26:43.512373+0000 (oldest deadline 2023-06-22T17:27:08.212584+0000) 2023-06-22T17:27:30.918 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:30 smithi039 conmon[115622]: 2023-06-22T17:27:30.457+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.015647+0000 front 2023-06-22T17:26:52.015497+0000 (oldest deadline 2023-06-22T17:27:19.016131+0000) 2023-06-22T17:27:30.918 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:30 smithi039 conmon[115622]: 2023-06-22T17:27:30.457+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:59.016552+0000 front 2023-06-22T17:26:59.016657+0000 (oldest deadline 2023-06-22T17:27:21.916722+0000) 2023-06-22T17:27:30.918 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:30 smithi039 conmon[115622]: 2023-06-22T17:27:30.457+0000 7fb4be092700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:05.318294+0000 front 2023-06-22T17:27:05.318270+0000 (oldest deadline 2023-06-22T17:27:29.418459+0000) 2023-06-22T17:27:31.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:30 smithi133 conmon[115049]: 2023-06-22T17:27:30.907+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:31.172 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:30 smithi133 conmon[115049]: 2023-06-22T17:27:30.907+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:31.172 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:30 smithi133 conmon[115049]: 2023-06-22T17:27:30.907+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:31.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:31 smithi133 conmon[107946]: 2023-06-22T17:27:31.001+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:31.173 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:31 smithi133 conmon[107946]: 2023-06-22T17:27:31.001+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.767866+0000 front 2023-06-22T17:26:52.767701+0000 (oldest deadline 2023-06-22T17:27:17.467693+0000) 2023-06-22T17:27:31.173 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:31 smithi133 conmon[107946]: 2023-06-22T17:27:31.001+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:59.168703+0000 front 2023-06-22T17:26:59.168748+0000 (oldest deadline 2023-06-22T17:27:21.468879+0000) 2023-06-22T17:27:31.173 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:31 smithi133 conmon[107946]: 2023-06-22T17:27:31.001+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.969905+0000 front 2023-06-22T17:27:04.970073+0000 (oldest deadline 2023-06-22T17:27:29.669924+0000) 2023-06-22T17:27:31.183 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:30 smithi039 conmon[122618]: 2023-06-22T17:27:30.915+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:31.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:30 smithi039 conmon[122618]: 2023-06-22T17:27:30.915+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:31.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:30 smithi039 conmon[122618]: 2023-06-22T17:27:30.915+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:27:00.343681+0000 front 2023-06-22T17:27:00.343658+0000 (oldest deadline 2023-06-22T17:27:21.443709+0000) 2023-06-22T17:27:31.785 INFO:journalctl@ceph.osd.6.smithi039.stdout:Jun 22 17:27:31 smithi039 bash[134626]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c-osd-6 2023-06-22T17:27:32.127 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:31 smithi101 conmon[105890]: 2023-06-22T17:27:31.626+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.932693+0000 front 2023-06-22T17:26:49.932674+0000 (oldest deadline 2023-06-22T17:27:15.232896+0000) 2023-06-22T17:27:32.127 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:31 smithi101 conmon[105890]: 2023-06-22T17:27:31.626+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:55.233427+0000 front 2023-06-22T17:26:55.233506+0000 (oldest deadline 2023-06-22T17:27:21.133441+0000) 2023-06-22T17:27:32.128 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:31 smithi101 conmon[105890]: 2023-06-22T17:27:31.626+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:06.434910+0000 front 2023-06-22T17:27:06.434807+0000 (oldest deadline 2023-06-22T17:27:28.734765+0000) 2023-06-22T17:27:32.128 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:31 smithi101 conmon[112983]: 2023-06-22T17:27:31.748+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:32.128 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:31 smithi101 conmon[112983]: 2023-06-22T17:27:31.748+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:32.129 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:31 smithi101 conmon[112983]: 2023-06-22T17:27:31.748+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:32.129 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:31 smithi101 conmon[112983]: 2023-06-22T17:27:31.748+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.163707+0000 front 2023-06-22T17:27:04.163629+0000 (oldest deadline 2023-06-22T17:27:30.564301+0000) 2023-06-22T17:27:32.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:31 smithi133 conmon[115049]: 2023-06-22T17:27:31.888+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:32.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:31 smithi133 conmon[115049]: 2023-06-22T17:27:31.888+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:32.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:31 smithi133 conmon[115049]: 2023-06-22T17:27:31.888+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:32.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:32 smithi133 conmon[107946]: 2023-06-22T17:27:32.035+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:32.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:32 smithi133 conmon[107946]: 2023-06-22T17:27:32.035+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.767866+0000 front 2023-06-22T17:26:52.767701+0000 (oldest deadline 2023-06-22T17:27:17.467693+0000) 2023-06-22T17:27:32.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:32 smithi133 conmon[107946]: 2023-06-22T17:27:32.035+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:59.168703+0000 front 2023-06-22T17:26:59.168748+0000 (oldest deadline 2023-06-22T17:27:21.468879+0000) 2023-06-22T17:27:32.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:32 smithi133 conmon[107946]: 2023-06-22T17:27:32.035+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.969905+0000 front 2023-06-22T17:27:04.970073+0000 (oldest deadline 2023-06-22T17:27:29.669924+0000) 2023-06-22T17:27:32.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:31 smithi039 conmon[122618]: 2023-06-22T17:27:31.877+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:32.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:31 smithi039 conmon[122618]: 2023-06-22T17:27:31.877+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:32.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:31 smithi039 conmon[122618]: 2023-06-22T17:27:31.877+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:27:00.343681+0000 front 2023-06-22T17:27:00.343658+0000 (oldest deadline 2023-06-22T17:27:21.443709+0000) 2023-06-22T17:27:32.185 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:31 smithi039 conmon[122618]: 2023-06-22T17:27:31.877+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:06.144593+0000 front 2023-06-22T17:27:06.144629+0000 (oldest deadline 2023-06-22T17:27:31.444781+0000) 2023-06-22T17:27:33.127 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:32 smithi101 conmon[105890]: 2023-06-22T17:27:32.638+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.932693+0000 front 2023-06-22T17:26:49.932674+0000 (oldest deadline 2023-06-22T17:27:15.232896+0000) 2023-06-22T17:27:33.127 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:32 smithi101 conmon[105890]: 2023-06-22T17:27:32.638+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:55.233427+0000 front 2023-06-22T17:26:55.233506+0000 (oldest deadline 2023-06-22T17:27:21.133441+0000) 2023-06-22T17:27:33.127 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:32 smithi101 conmon[105890]: 2023-06-22T17:27:32.638+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:06.434910+0000 front 2023-06-22T17:27:06.434807+0000 (oldest deadline 2023-06-22T17:27:28.734765+0000) 2023-06-22T17:27:33.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:32 smithi101 conmon[112983]: 2023-06-22T17:27:32.752+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:33.128 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:32 smithi101 conmon[112983]: 2023-06-22T17:27:32.752+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:33.128 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:32 smithi101 conmon[112983]: 2023-06-22T17:27:32.752+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:33.128 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:32 smithi101 conmon[112983]: 2023-06-22T17:27:32.752+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.163707+0000 front 2023-06-22T17:27:04.163629+0000 (oldest deadline 2023-06-22T17:27:30.564301+0000) 2023-06-22T17:27:33.154 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:32 smithi039 conmon[122618]: 2023-06-22T17:27:32.829+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:33.154 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:32 smithi039 conmon[122618]: 2023-06-22T17:27:32.829+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:33.154 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:32 smithi039 conmon[122618]: 2023-06-22T17:27:32.829+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:27:00.343681+0000 front 2023-06-22T17:27:00.343658+0000 (oldest deadline 2023-06-22T17:27:21.443709+0000) 2023-06-22T17:27:33.155 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:32 smithi039 conmon[122618]: 2023-06-22T17:27:32.829+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:06.144593+0000 front 2023-06-22T17:27:06.144629+0000 (oldest deadline 2023-06-22T17:27:31.444781+0000) 2023-06-22T17:27:33.162 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.6.service' 2023-06-22T17:27:33.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:32 smithi133 conmon[115049]: 2023-06-22T17:27:32.889+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:33.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:32 smithi133 conmon[115049]: 2023-06-22T17:27:32.889+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:33.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:32 smithi133 conmon[115049]: 2023-06-22T17:27:32.889+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:33.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:33 smithi133 conmon[107946]: 2023-06-22T17:27:33.021+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:33.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:33 smithi133 conmon[107946]: 2023-06-22T17:27:33.021+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.767866+0000 front 2023-06-22T17:26:52.767701+0000 (oldest deadline 2023-06-22T17:27:17.467693+0000) 2023-06-22T17:27:33.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:33 smithi133 conmon[107946]: 2023-06-22T17:27:33.021+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:59.168703+0000 front 2023-06-22T17:26:59.168748+0000 (oldest deadline 2023-06-22T17:27:21.468879+0000) 2023-06-22T17:27:33.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:33 smithi133 conmon[107946]: 2023-06-22T17:27:33.021+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.969905+0000 front 2023-06-22T17:27:04.970073+0000 (oldest deadline 2023-06-22T17:27:29.669924+0000) 2023-06-22T17:27:33.241 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-22T17:27:33.242 INFO:tasks.cephadm.osd.6:Stopped osd.6 2023-06-22T17:27:33.242 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2023-06-22T17:27:33.243 DEBUG:teuthology.orchestra.run.smithi101:> sudo systemctl stop ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.7 2023-06-22T17:27:33.618 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:33 smithi101 systemd[1]: Stopping Ceph osd.7 for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:27:33.877 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:33 smithi101 conmon[112983]: 2023-06-22T17:27:33.784+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:33.877 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:33 smithi101 conmon[112983]: 2023-06-22T17:27:33.784+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:33.877 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:33 smithi101 conmon[112983]: 2023-06-22T17:27:33.784+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:33.878 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:33 smithi101 conmon[112983]: 2023-06-22T17:27:33.784+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.163707+0000 front 2023-06-22T17:27:04.163629+0000 (oldest deadline 2023-06-22T17:27:30.564301+0000) 2023-06-22T17:27:33.878 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:33 smithi101 conmon[105890]: 2023-06-22T17:27:33.615+0000 7f2712024700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-22T17:27:33.878 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:33 smithi101 conmon[105890]: 2023-06-22T17:27:33.615+0000 7f2712024700 -1 osd.7 81 *** Got signal Terminated *** 2023-06-22T17:27:33.878 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:33 smithi101 conmon[105890]: 2023-06-22T17:27:33.615+0000 7f2712024700 -1 osd.7 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-06-22T17:27:33.878 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:33 smithi101 conmon[105890]: 2023-06-22T17:27:33.629+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.932693+0000 front 2023-06-22T17:26:49.932674+0000 (oldest deadline 2023-06-22T17:27:15.232896+0000) 2023-06-22T17:27:33.878 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:33 smithi101 conmon[105890]: 2023-06-22T17:27:33.629+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:55.233427+0000 front 2023-06-22T17:26:55.233506+0000 (oldest deadline 2023-06-22T17:27:21.133441+0000) 2023-06-22T17:27:33.879 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:33 smithi101 conmon[105890]: 2023-06-22T17:27:33.629+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:06.434910+0000 front 2023-06-22T17:27:06.434807+0000 (oldest deadline 2023-06-22T17:27:28.734765+0000) 2023-06-22T17:27:34.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:33 smithi039 conmon[122618]: 2023-06-22T17:27:33.821+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:34.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:33 smithi039 conmon[122618]: 2023-06-22T17:27:33.821+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:34.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:33 smithi039 conmon[122618]: 2023-06-22T17:27:33.821+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:27:00.343681+0000 front 2023-06-22T17:27:00.343658+0000 (oldest deadline 2023-06-22T17:27:21.443709+0000) 2023-06-22T17:27:34.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:33 smithi039 conmon[122618]: 2023-06-22T17:27:33.821+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:06.144593+0000 front 2023-06-22T17:27:06.144629+0000 (oldest deadline 2023-06-22T17:27:31.444781+0000) 2023-06-22T17:27:34.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:33 smithi133 conmon[115049]: 2023-06-22T17:27:33.929+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:34.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:33 smithi133 conmon[115049]: 2023-06-22T17:27:33.929+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:34.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:33 smithi133 conmon[115049]: 2023-06-22T17:27:33.929+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:34.422 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:34 smithi133 conmon[107946]: 2023-06-22T17:27:34.021+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:34.422 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:34 smithi133 conmon[107946]: 2023-06-22T17:27:34.021+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.767866+0000 front 2023-06-22T17:26:52.767701+0000 (oldest deadline 2023-06-22T17:27:17.467693+0000) 2023-06-22T17:27:34.422 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:34 smithi133 conmon[107946]: 2023-06-22T17:27:34.021+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:59.168703+0000 front 2023-06-22T17:26:59.168748+0000 (oldest deadline 2023-06-22T17:27:21.468879+0000) 2023-06-22T17:27:34.422 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:34 smithi133 conmon[107946]: 2023-06-22T17:27:34.021+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.969905+0000 front 2023-06-22T17:27:04.970073+0000 (oldest deadline 2023-06-22T17:27:29.669924+0000) 2023-06-22T17:27:35.127 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:34 smithi101 conmon[105890]: 2023-06-22T17:27:34.627+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.932693+0000 front 2023-06-22T17:26:49.932674+0000 (oldest deadline 2023-06-22T17:27:15.232896+0000) 2023-06-22T17:27:35.127 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:34 smithi101 conmon[105890]: 2023-06-22T17:27:34.627+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:55.233427+0000 front 2023-06-22T17:26:55.233506+0000 (oldest deadline 2023-06-22T17:27:21.133441+0000) 2023-06-22T17:27:35.127 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:34 smithi101 conmon[105890]: 2023-06-22T17:27:34.627+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:06.434910+0000 front 2023-06-22T17:27:06.434807+0000 (oldest deadline 2023-06-22T17:27:28.734765+0000) 2023-06-22T17:27:35.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:34 smithi101 conmon[112983]: 2023-06-22T17:27:34.772+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:35.128 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:34 smithi101 conmon[112983]: 2023-06-22T17:27:34.772+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:35.128 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:34 smithi101 conmon[112983]: 2023-06-22T17:27:34.772+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:35.128 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:34 smithi101 conmon[112983]: 2023-06-22T17:27:34.772+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.163707+0000 front 2023-06-22T17:27:04.163629+0000 (oldest deadline 2023-06-22T17:27:30.564301+0000) 2023-06-22T17:27:35.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:34 smithi133 conmon[115049]: 2023-06-22T17:27:34.902+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:35.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:34 smithi133 conmon[115049]: 2023-06-22T17:27:34.902+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:35.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:34 smithi133 conmon[115049]: 2023-06-22T17:27:34.902+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:35.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:34 smithi133 conmon[107946]: 2023-06-22T17:27:34.994+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:35.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:34 smithi133 conmon[107946]: 2023-06-22T17:27:34.994+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.767866+0000 front 2023-06-22T17:26:52.767701+0000 (oldest deadline 2023-06-22T17:27:17.467693+0000) 2023-06-22T17:27:35.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:34 smithi133 conmon[107946]: 2023-06-22T17:27:34.994+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:59.168703+0000 front 2023-06-22T17:26:59.168748+0000 (oldest deadline 2023-06-22T17:27:21.468879+0000) 2023-06-22T17:27:35.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:34 smithi133 conmon[107946]: 2023-06-22T17:27:34.994+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.969905+0000 front 2023-06-22T17:27:04.970073+0000 (oldest deadline 2023-06-22T17:27:29.669924+0000) 2023-06-22T17:27:35.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:34 smithi039 conmon[122618]: 2023-06-22T17:27:34.784+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:35.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:34 smithi039 conmon[122618]: 2023-06-22T17:27:34.784+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:35.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:34 smithi039 conmon[122618]: 2023-06-22T17:27:34.784+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:27:00.343681+0000 front 2023-06-22T17:27:00.343658+0000 (oldest deadline 2023-06-22T17:27:21.443709+0000) 2023-06-22T17:27:35.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:34 smithi039 conmon[122618]: 2023-06-22T17:27:34.784+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:06.144593+0000 front 2023-06-22T17:27:06.144629+0000 (oldest deadline 2023-06-22T17:27:31.444781+0000) 2023-06-22T17:27:36.127 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:35 smithi101 conmon[105890]: 2023-06-22T17:27:35.639+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.932693+0000 front 2023-06-22T17:26:49.932674+0000 (oldest deadline 2023-06-22T17:27:15.232896+0000) 2023-06-22T17:27:36.127 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:35 smithi101 conmon[105890]: 2023-06-22T17:27:35.639+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:55.233427+0000 front 2023-06-22T17:26:55.233506+0000 (oldest deadline 2023-06-22T17:27:21.133441+0000) 2023-06-22T17:27:36.127 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:35 smithi101 conmon[105890]: 2023-06-22T17:27:35.639+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:06.434910+0000 front 2023-06-22T17:27:06.434807+0000 (oldest deadline 2023-06-22T17:27:28.734765+0000) 2023-06-22T17:27:36.128 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:35 smithi101 conmon[112983]: 2023-06-22T17:27:35.761+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:36.128 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:35 smithi101 conmon[112983]: 2023-06-22T17:27:35.761+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:36.128 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:35 smithi101 conmon[112983]: 2023-06-22T17:27:35.761+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:36.128 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:35 smithi101 conmon[112983]: 2023-06-22T17:27:35.761+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.163707+0000 front 2023-06-22T17:27:04.163629+0000 (oldest deadline 2023-06-22T17:27:30.564301+0000) 2023-06-22T17:27:36.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:35 smithi133 conmon[115049]: 2023-06-22T17:27:35.860+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:36.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:35 smithi133 conmon[115049]: 2023-06-22T17:27:35.860+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:36.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:35 smithi133 conmon[115049]: 2023-06-22T17:27:35.860+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:36.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:35 smithi133 conmon[107946]: 2023-06-22T17:27:35.963+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:36.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:35 smithi133 conmon[107946]: 2023-06-22T17:27:35.963+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.767866+0000 front 2023-06-22T17:26:52.767701+0000 (oldest deadline 2023-06-22T17:27:17.467693+0000) 2023-06-22T17:27:36.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:35 smithi133 conmon[107946]: 2023-06-22T17:27:35.963+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:59.168703+0000 front 2023-06-22T17:26:59.168748+0000 (oldest deadline 2023-06-22T17:27:21.468879+0000) 2023-06-22T17:27:36.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:35 smithi133 conmon[107946]: 2023-06-22T17:27:35.963+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.969905+0000 front 2023-06-22T17:27:04.970073+0000 (oldest deadline 2023-06-22T17:27:29.669924+0000) 2023-06-22T17:27:36.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:35 smithi039 conmon[122618]: 2023-06-22T17:27:35.788+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:36.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:35 smithi039 conmon[122618]: 2023-06-22T17:27:35.788+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:36.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:35 smithi039 conmon[122618]: 2023-06-22T17:27:35.788+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:27:00.343681+0000 front 2023-06-22T17:27:00.343658+0000 (oldest deadline 2023-06-22T17:27:21.443709+0000) 2023-06-22T17:27:36.185 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:35 smithi039 conmon[122618]: 2023-06-22T17:27:35.788+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:06.144593+0000 front 2023-06-22T17:27:06.144629+0000 (oldest deadline 2023-06-22T17:27:31.444781+0000) 2023-06-22T17:27:36.877 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:36 smithi101 conmon[105890]: 2023-06-22T17:27:36.623+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.932693+0000 front 2023-06-22T17:26:49.932674+0000 (oldest deadline 2023-06-22T17:27:15.232896+0000) 2023-06-22T17:27:36.877 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:36 smithi101 conmon[105890]: 2023-06-22T17:27:36.623+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:55.233427+0000 front 2023-06-22T17:26:55.233506+0000 (oldest deadline 2023-06-22T17:27:21.133441+0000) 2023-06-22T17:27:36.877 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:36 smithi101 conmon[105890]: 2023-06-22T17:27:36.623+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:06.434910+0000 front 2023-06-22T17:27:06.434807+0000 (oldest deadline 2023-06-22T17:27:28.734765+0000) 2023-06-22T17:27:36.877 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:36 smithi101 conmon[105890]: 2023-06-22T17:27:36.623+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:10.435604+0000 front 2023-06-22T17:27:10.435650+0000 (oldest deadline 2023-06-22T17:27:36.335753+0000) 2023-06-22T17:27:36.878 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:36 smithi101 conmon[112983]: 2023-06-22T17:27:36.775+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:36.878 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:36 smithi101 conmon[112983]: 2023-06-22T17:27:36.775+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:36.878 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:36 smithi101 conmon[112983]: 2023-06-22T17:27:36.775+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:36.878 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:36 smithi101 conmon[112983]: 2023-06-22T17:27:36.775+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.163707+0000 front 2023-06-22T17:27:04.163629+0000 (oldest deadline 2023-06-22T17:27:30.564301+0000) 2023-06-22T17:27:37.171 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:37 smithi133 conmon[107946]: 2023-06-22T17:27:37.000+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:37.171 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:37 smithi133 conmon[107946]: 2023-06-22T17:27:37.000+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.767866+0000 front 2023-06-22T17:26:52.767701+0000 (oldest deadline 2023-06-22T17:27:17.467693+0000) 2023-06-22T17:27:37.171 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:37 smithi133 conmon[107946]: 2023-06-22T17:27:37.000+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:59.168703+0000 front 2023-06-22T17:26:59.168748+0000 (oldest deadline 2023-06-22T17:27:21.468879+0000) 2023-06-22T17:27:37.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:37 smithi133 conmon[107946]: 2023-06-22T17:27:37.000+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.969905+0000 front 2023-06-22T17:27:04.970073+0000 (oldest deadline 2023-06-22T17:27:29.669924+0000) 2023-06-22T17:27:37.172 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:36 smithi133 conmon[115049]: 2023-06-22T17:27:36.840+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:37.172 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:36 smithi133 conmon[115049]: 2023-06-22T17:27:36.840+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:37.172 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:36 smithi133 conmon[115049]: 2023-06-22T17:27:36.840+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:37.172 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:36 smithi133 conmon[115049]: 2023-06-22T17:27:36.840+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:13.936567+0000 front 2023-06-22T17:27:13.936510+0000 (oldest deadline 2023-06-22T17:27:36.236444+0000) 2023-06-22T17:27:37.183 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:36 smithi039 conmon[122618]: 2023-06-22T17:27:36.756+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:37.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:36 smithi039 conmon[122618]: 2023-06-22T17:27:36.756+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:37.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:36 smithi039 conmon[122618]: 2023-06-22T17:27:36.756+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:27:00.343681+0000 front 2023-06-22T17:27:00.343658+0000 (oldest deadline 2023-06-22T17:27:21.443709+0000) 2023-06-22T17:27:37.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:36 smithi039 conmon[122618]: 2023-06-22T17:27:36.756+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:06.144593+0000 front 2023-06-22T17:27:06.144629+0000 (oldest deadline 2023-06-22T17:27:31.444781+0000) 2023-06-22T17:27:37.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:36 smithi039 conmon[122618]: 2023-06-22T17:27:36.756+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:11.445285+0000 front 2023-06-22T17:27:11.445252+0000 (oldest deadline 2023-06-22T17:27:36.145376+0000) 2023-06-22T17:27:37.877 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:37 smithi101 conmon[105890]: 2023-06-22T17:27:37.610+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.932693+0000 front 2023-06-22T17:26:49.932674+0000 (oldest deadline 2023-06-22T17:27:15.232896+0000) 2023-06-22T17:27:37.877 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:37 smithi101 conmon[105890]: 2023-06-22T17:27:37.610+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:55.233427+0000 front 2023-06-22T17:26:55.233506+0000 (oldest deadline 2023-06-22T17:27:21.133441+0000) 2023-06-22T17:27:37.877 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:37 smithi101 conmon[105890]: 2023-06-22T17:27:37.610+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:06.434910+0000 front 2023-06-22T17:27:06.434807+0000 (oldest deadline 2023-06-22T17:27:28.734765+0000) 2023-06-22T17:27:37.878 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:37 smithi101 conmon[105890]: 2023-06-22T17:27:37.610+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:10.435604+0000 front 2023-06-22T17:27:10.435650+0000 (oldest deadline 2023-06-22T17:27:36.335753+0000) 2023-06-22T17:27:37.878 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:37 smithi101 conmon[112983]: 2023-06-22T17:27:37.825+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:37.878 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:37 smithi101 conmon[112983]: 2023-06-22T17:27:37.825+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:37.879 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:37 smithi101 conmon[112983]: 2023-06-22T17:27:37.825+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:37.879 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:37 smithi101 conmon[112983]: 2023-06-22T17:27:37.825+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.163707+0000 front 2023-06-22T17:27:04.163629+0000 (oldest deadline 2023-06-22T17:27:30.564301+0000) 2023-06-22T17:27:38.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:37 smithi133 conmon[115049]: 2023-06-22T17:27:37.874+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:38.172 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:37 smithi133 conmon[115049]: 2023-06-22T17:27:37.874+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:38.172 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:37 smithi133 conmon[115049]: 2023-06-22T17:27:37.874+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:38.172 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:37 smithi133 conmon[115049]: 2023-06-22T17:27:37.874+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:13.936567+0000 front 2023-06-22T17:27:13.936510+0000 (oldest deadline 2023-06-22T17:27:36.236444+0000) 2023-06-22T17:27:38.173 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:38 smithi133 conmon[107946]: 2023-06-22T17:27:38.011+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:38.173 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:38 smithi133 conmon[107946]: 2023-06-22T17:27:38.011+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.767866+0000 front 2023-06-22T17:26:52.767701+0000 (oldest deadline 2023-06-22T17:27:17.467693+0000) 2023-06-22T17:27:38.173 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:38 smithi133 conmon[107946]: 2023-06-22T17:27:38.011+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:59.168703+0000 front 2023-06-22T17:26:59.168748+0000 (oldest deadline 2023-06-22T17:27:21.468879+0000) 2023-06-22T17:27:38.173 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:38 smithi133 conmon[107946]: 2023-06-22T17:27:38.011+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.969905+0000 front 2023-06-22T17:27:04.970073+0000 (oldest deadline 2023-06-22T17:27:29.669924+0000) 2023-06-22T17:27:38.174 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:38 smithi133 conmon[107946]: 2023-06-22T17:27:38.011+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:14.172382+0000 front 2023-06-22T17:27:14.172429+0000 (oldest deadline 2023-06-22T17:27:37.072457+0000) 2023-06-22T17:27:38.183 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:37 smithi039 conmon[122618]: 2023-06-22T17:27:37.732+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:38.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:37 smithi039 conmon[122618]: 2023-06-22T17:27:37.732+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:38.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:37 smithi039 conmon[122618]: 2023-06-22T17:27:37.732+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:27:00.343681+0000 front 2023-06-22T17:27:00.343658+0000 (oldest deadline 2023-06-22T17:27:21.443709+0000) 2023-06-22T17:27:38.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:37 smithi039 conmon[122618]: 2023-06-22T17:27:37.732+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:06.144593+0000 front 2023-06-22T17:27:06.144629+0000 (oldest deadline 2023-06-22T17:27:31.444781+0000) 2023-06-22T17:27:38.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:37 smithi039 conmon[122618]: 2023-06-22T17:27:37.732+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:11.445285+0000 front 2023-06-22T17:27:11.445252+0000 (oldest deadline 2023-06-22T17:27:36.145376+0000) 2023-06-22T17:27:38.858 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:38 smithi101 conmon[105890]: 2023-06-22T17:27:38.580+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.932693+0000 front 2023-06-22T17:26:49.932674+0000 (oldest deadline 2023-06-22T17:27:15.232896+0000) 2023-06-22T17:27:38.858 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:38 smithi101 conmon[105890]: 2023-06-22T17:27:38.580+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:55.233427+0000 front 2023-06-22T17:26:55.233506+0000 (oldest deadline 2023-06-22T17:27:21.133441+0000) 2023-06-22T17:27:38.858 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:38 smithi101 conmon[105890]: 2023-06-22T17:27:38.580+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:06.434910+0000 front 2023-06-22T17:27:06.434807+0000 (oldest deadline 2023-06-22T17:27:28.734765+0000) 2023-06-22T17:27:38.858 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:38 smithi101 conmon[105890]: 2023-06-22T17:27:38.580+0000 7f270d620700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:10.435604+0000 front 2023-06-22T17:27:10.435650+0000 (oldest deadline 2023-06-22T17:27:36.335753+0000) 2023-06-22T17:27:39.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:38 smithi101 conmon[112983]: 2023-06-22T17:27:38.855+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:39.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:38 smithi101 conmon[112983]: 2023-06-22T17:27:38.855+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:39.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:38 smithi101 conmon[112983]: 2023-06-22T17:27:38.855+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:39.128 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:38 smithi101 conmon[112983]: 2023-06-22T17:27:38.855+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.163707+0000 front 2023-06-22T17:27:04.163629+0000 (oldest deadline 2023-06-22T17:27:30.564301+0000) 2023-06-22T17:27:39.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:38 smithi133 conmon[115049]: 2023-06-22T17:27:38.828+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:39.172 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:38 smithi133 conmon[115049]: 2023-06-22T17:27:38.828+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:39.172 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:38 smithi133 conmon[115049]: 2023-06-22T17:27:38.828+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:39.172 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:38 smithi133 conmon[115049]: 2023-06-22T17:27:38.828+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:13.936567+0000 front 2023-06-22T17:27:13.936510+0000 (oldest deadline 2023-06-22T17:27:36.236444+0000) 2023-06-22T17:27:39.173 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:38 smithi133 conmon[107946]: 2023-06-22T17:27:38.970+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:39.173 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:38 smithi133 conmon[107946]: 2023-06-22T17:27:38.970+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.767866+0000 front 2023-06-22T17:26:52.767701+0000 (oldest deadline 2023-06-22T17:27:17.467693+0000) 2023-06-22T17:27:39.173 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:38 smithi133 conmon[107946]: 2023-06-22T17:27:38.970+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:59.168703+0000 front 2023-06-22T17:26:59.168748+0000 (oldest deadline 2023-06-22T17:27:21.468879+0000) 2023-06-22T17:27:39.173 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:38 smithi133 conmon[107946]: 2023-06-22T17:27:38.970+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.969905+0000 front 2023-06-22T17:27:04.970073+0000 (oldest deadline 2023-06-22T17:27:29.669924+0000) 2023-06-22T17:27:39.174 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:38 smithi133 conmon[107946]: 2023-06-22T17:27:38.970+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:14.172382+0000 front 2023-06-22T17:27:14.172429+0000 (oldest deadline 2023-06-22T17:27:37.072457+0000) 2023-06-22T17:27:39.183 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:38 smithi039 conmon[122618]: 2023-06-22T17:27:38.762+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:39.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:38 smithi039 conmon[122618]: 2023-06-22T17:27:38.762+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:39.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:38 smithi039 conmon[122618]: 2023-06-22T17:27:38.762+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:27:00.343681+0000 front 2023-06-22T17:27:00.343658+0000 (oldest deadline 2023-06-22T17:27:21.443709+0000) 2023-06-22T17:27:39.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:38 smithi039 conmon[122618]: 2023-06-22T17:27:38.762+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:06.144593+0000 front 2023-06-22T17:27:06.144629+0000 (oldest deadline 2023-06-22T17:27:31.444781+0000) 2023-06-22T17:27:39.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:38 smithi039 conmon[122618]: 2023-06-22T17:27:38.762+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:11.445285+0000 front 2023-06-22T17:27:11.445252+0000 (oldest deadline 2023-06-22T17:27:36.145376+0000) 2023-06-22T17:27:39.444 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jun 22 17:27:39 smithi101 bash[122279]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c-osd-7 2023-06-22T17:27:40.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:39 smithi101 conmon[112983]: 2023-06-22T17:27:39.843+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:40.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:39 smithi101 conmon[112983]: 2023-06-22T17:27:39.843+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:40.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:39 smithi101 conmon[112983]: 2023-06-22T17:27:39.843+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:40.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:39 smithi101 conmon[112983]: 2023-06-22T17:27:39.843+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.163707+0000 front 2023-06-22T17:27:04.163629+0000 (oldest deadline 2023-06-22T17:27:30.564301+0000) 2023-06-22T17:27:40.128 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:39 smithi101 conmon[112983]: 2023-06-22T17:27:39.843+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:14.065051+0000 front 2023-06-22T17:27:14.065033+0000 (oldest deadline 2023-06-22T17:27:39.365200+0000) 2023-06-22T17:27:40.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:39 smithi133 conmon[115049]: 2023-06-22T17:27:39.784+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:40.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:39 smithi133 conmon[115049]: 2023-06-22T17:27:39.784+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:40.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:39 smithi133 conmon[115049]: 2023-06-22T17:27:39.784+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:40.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:39 smithi133 conmon[115049]: 2023-06-22T17:27:39.784+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:13.936567+0000 front 2023-06-22T17:27:13.936510+0000 (oldest deadline 2023-06-22T17:27:36.236444+0000) 2023-06-22T17:27:40.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:39 smithi133 conmon[107946]: 2023-06-22T17:27:39.992+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:40.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:39 smithi133 conmon[107946]: 2023-06-22T17:27:39.992+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.767866+0000 front 2023-06-22T17:26:52.767701+0000 (oldest deadline 2023-06-22T17:27:17.467693+0000) 2023-06-22T17:27:40.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:39 smithi133 conmon[107946]: 2023-06-22T17:27:39.992+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:59.168703+0000 front 2023-06-22T17:26:59.168748+0000 (oldest deadline 2023-06-22T17:27:21.468879+0000) 2023-06-22T17:27:40.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:39 smithi133 conmon[107946]: 2023-06-22T17:27:39.992+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.969905+0000 front 2023-06-22T17:27:04.970073+0000 (oldest deadline 2023-06-22T17:27:29.669924+0000) 2023-06-22T17:27:40.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:39 smithi133 conmon[107946]: 2023-06-22T17:27:39.992+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:14.172382+0000 front 2023-06-22T17:27:14.172429+0000 (oldest deadline 2023-06-22T17:27:37.072457+0000) 2023-06-22T17:27:40.183 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:39 smithi039 conmon[122618]: 2023-06-22T17:27:39.768+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:40.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:39 smithi039 conmon[122618]: 2023-06-22T17:27:39.768+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:40.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:39 smithi039 conmon[122618]: 2023-06-22T17:27:39.768+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:27:00.343681+0000 front 2023-06-22T17:27:00.343658+0000 (oldest deadline 2023-06-22T17:27:21.443709+0000) 2023-06-22T17:27:40.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:39 smithi039 conmon[122618]: 2023-06-22T17:27:39.768+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:06.144593+0000 front 2023-06-22T17:27:06.144629+0000 (oldest deadline 2023-06-22T17:27:31.444781+0000) 2023-06-22T17:27:40.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:39 smithi039 conmon[122618]: 2023-06-22T17:27:39.768+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:11.445285+0000 front 2023-06-22T17:27:11.445252+0000 (oldest deadline 2023-06-22T17:27:36.145376+0000) 2023-06-22T17:27:40.801 DEBUG:teuthology.orchestra.run.smithi101:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.7.service' 2023-06-22T17:27:40.839 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-22T17:27:40.840 INFO:tasks.cephadm.osd.7:Stopped osd.7 2023-06-22T17:27:40.840 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2023-06-22T17:27:40.841 DEBUG:teuthology.orchestra.run.smithi133:> sudo systemctl stop ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.8 2023-06-22T17:27:41.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:40 smithi101 conmon[112983]: 2023-06-22T17:27:40.879+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:41.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:40 smithi101 conmon[112983]: 2023-06-22T17:27:40.880+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:41.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:40 smithi101 conmon[112983]: 2023-06-22T17:27:40.880+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:41.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:40 smithi101 conmon[112983]: 2023-06-22T17:27:40.880+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.163707+0000 front 2023-06-22T17:27:04.163629+0000 (oldest deadline 2023-06-22T17:27:30.564301+0000) 2023-06-22T17:27:41.128 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:40 smithi101 conmon[112983]: 2023-06-22T17:27:40.880+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:14.065051+0000 front 2023-06-22T17:27:14.065033+0000 (oldest deadline 2023-06-22T17:27:39.365200+0000) 2023-06-22T17:27:41.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:40 smithi133 systemd[1]: Stopping Ceph osd.8 for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:27:41.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:41 smithi133 conmon[107946]: 2023-06-22T17:27:41.036+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:41.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:41 smithi133 conmon[107946]: 2023-06-22T17:27:41.036+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.767866+0000 front 2023-06-22T17:26:52.767701+0000 (oldest deadline 2023-06-22T17:27:17.467693+0000) 2023-06-22T17:27:41.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:41 smithi133 conmon[107946]: 2023-06-22T17:27:41.036+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:59.168703+0000 front 2023-06-22T17:26:59.168748+0000 (oldest deadline 2023-06-22T17:27:21.468879+0000) 2023-06-22T17:27:41.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:41 smithi133 conmon[107946]: 2023-06-22T17:27:41.036+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.969905+0000 front 2023-06-22T17:27:04.970073+0000 (oldest deadline 2023-06-22T17:27:29.669924+0000) 2023-06-22T17:27:41.173 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:41 smithi133 conmon[107946]: 2023-06-22T17:27:41.036+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:14.172382+0000 front 2023-06-22T17:27:14.172429+0000 (oldest deadline 2023-06-22T17:27:37.072457+0000) 2023-06-22T17:27:41.173 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:40 smithi133 conmon[115049]: 2023-06-22T17:27:40.830+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:41.173 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:40 smithi133 conmon[115049]: 2023-06-22T17:27:40.830+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:41.173 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:40 smithi133 conmon[115049]: 2023-06-22T17:27:40.830+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:41.173 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:40 smithi133 conmon[115049]: 2023-06-22T17:27:40.830+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:13.936567+0000 front 2023-06-22T17:27:13.936510+0000 (oldest deadline 2023-06-22T17:27:36.236444+0000) 2023-06-22T17:27:41.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:40 smithi039 conmon[122618]: 2023-06-22T17:27:40.731+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:41.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:40 smithi039 conmon[122618]: 2023-06-22T17:27:40.731+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:41.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:40 smithi039 conmon[122618]: 2023-06-22T17:27:40.731+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:27:00.343681+0000 front 2023-06-22T17:27:00.343658+0000 (oldest deadline 2023-06-22T17:27:21.443709+0000) 2023-06-22T17:27:41.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:40 smithi039 conmon[122618]: 2023-06-22T17:27:40.731+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:06.144593+0000 front 2023-06-22T17:27:06.144629+0000 (oldest deadline 2023-06-22T17:27:31.444781+0000) 2023-06-22T17:27:41.185 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:40 smithi039 conmon[122618]: 2023-06-22T17:27:40.731+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:11.445285+0000 front 2023-06-22T17:27:11.445252+0000 (oldest deadline 2023-06-22T17:27:36.145376+0000) 2023-06-22T17:27:41.671 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:41 smithi133 conmon[107946]: 2023-06-22T17:27:41.206+0000 7f717b5c0700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-22T17:27:41.671 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:41 smithi133 conmon[107946]: 2023-06-22T17:27:41.206+0000 7f717b5c0700 -1 osd.8 81 *** Got signal Terminated *** 2023-06-22T17:27:41.671 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:41 smithi133 conmon[107946]: 2023-06-22T17:27:41.206+0000 7f717b5c0700 -1 osd.8 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-06-22T17:27:42.171 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:41 smithi133 conmon[107946]: 2023-06-22T17:27:41.992+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:42.171 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:41 smithi133 conmon[107946]: 2023-06-22T17:27:41.992+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.767866+0000 front 2023-06-22T17:26:52.767701+0000 (oldest deadline 2023-06-22T17:27:17.467693+0000) 2023-06-22T17:27:42.171 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:41 smithi133 conmon[107946]: 2023-06-22T17:27:41.992+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:59.168703+0000 front 2023-06-22T17:26:59.168748+0000 (oldest deadline 2023-06-22T17:27:21.468879+0000) 2023-06-22T17:27:42.171 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:41 smithi133 conmon[107946]: 2023-06-22T17:27:41.992+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.969905+0000 front 2023-06-22T17:27:04.970073+0000 (oldest deadline 2023-06-22T17:27:29.669924+0000) 2023-06-22T17:27:42.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:41 smithi133 conmon[107946]: 2023-06-22T17:27:41.992+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:14.172382+0000 front 2023-06-22T17:27:14.172429+0000 (oldest deadline 2023-06-22T17:27:37.072457+0000) 2023-06-22T17:27:42.172 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:41 smithi133 conmon[115049]: 2023-06-22T17:27:41.843+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:42.172 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:41 smithi133 conmon[115049]: 2023-06-22T17:27:41.843+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:42.172 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:41 smithi133 conmon[115049]: 2023-06-22T17:27:41.843+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:42.173 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:41 smithi133 conmon[115049]: 2023-06-22T17:27:41.843+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:13.936567+0000 front 2023-06-22T17:27:13.936510+0000 (oldest deadline 2023-06-22T17:27:36.236444+0000) 2023-06-22T17:27:42.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:41 smithi039 conmon[122618]: 2023-06-22T17:27:41.703+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:42.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:41 smithi039 conmon[122618]: 2023-06-22T17:27:41.703+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:42.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:41 smithi039 conmon[122618]: 2023-06-22T17:27:41.703+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:27:00.343681+0000 front 2023-06-22T17:27:00.343658+0000 (oldest deadline 2023-06-22T17:27:21.443709+0000) 2023-06-22T17:27:42.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:41 smithi039 conmon[122618]: 2023-06-22T17:27:41.703+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:06.144593+0000 front 2023-06-22T17:27:06.144629+0000 (oldest deadline 2023-06-22T17:27:31.444781+0000) 2023-06-22T17:27:42.185 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:41 smithi039 conmon[122618]: 2023-06-22T17:27:41.703+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:11.445285+0000 front 2023-06-22T17:27:11.445252+0000 (oldest deadline 2023-06-22T17:27:36.145376+0000) 2023-06-22T17:27:42.376 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:41 smithi101 conmon[112983]: 2023-06-22T17:27:41.887+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:42.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:41 smithi101 conmon[112983]: 2023-06-22T17:27:41.887+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:42.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:41 smithi101 conmon[112983]: 2023-06-22T17:27:41.887+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:42.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:41 smithi101 conmon[112983]: 2023-06-22T17:27:41.887+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.163707+0000 front 2023-06-22T17:27:04.163629+0000 (oldest deadline 2023-06-22T17:27:30.564301+0000) 2023-06-22T17:27:42.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:41 smithi101 conmon[112983]: 2023-06-22T17:27:41.887+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:14.065051+0000 front 2023-06-22T17:27:14.065033+0000 (oldest deadline 2023-06-22T17:27:39.365200+0000) 2023-06-22T17:27:42.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:42 smithi039 conmon[122618]: 2023-06-22T17:27:42.677+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:42.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:42 smithi039 conmon[122618]: 2023-06-22T17:27:42.677+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:42.935 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:42 smithi039 conmon[122618]: 2023-06-22T17:27:42.677+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:27:00.343681+0000 front 2023-06-22T17:27:00.343658+0000 (oldest deadline 2023-06-22T17:27:21.443709+0000) 2023-06-22T17:27:42.935 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:42 smithi039 conmon[122618]: 2023-06-22T17:27:42.677+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:06.144593+0000 front 2023-06-22T17:27:06.144629+0000 (oldest deadline 2023-06-22T17:27:31.444781+0000) 2023-06-22T17:27:42.935 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:42 smithi039 conmon[122618]: 2023-06-22T17:27:42.677+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:11.445285+0000 front 2023-06-22T17:27:11.445252+0000 (oldest deadline 2023-06-22T17:27:36.145376+0000) 2023-06-22T17:27:43.171 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:42 smithi133 conmon[107946]: 2023-06-22T17:27:42.959+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:43.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:42 smithi133 conmon[107946]: 2023-06-22T17:27:42.959+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.767866+0000 front 2023-06-22T17:26:52.767701+0000 (oldest deadline 2023-06-22T17:27:17.467693+0000) 2023-06-22T17:27:43.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:42 smithi133 conmon[107946]: 2023-06-22T17:27:42.959+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:59.168703+0000 front 2023-06-22T17:26:59.168748+0000 (oldest deadline 2023-06-22T17:27:21.468879+0000) 2023-06-22T17:27:43.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:42 smithi133 conmon[107946]: 2023-06-22T17:27:42.959+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.969905+0000 front 2023-06-22T17:27:04.970073+0000 (oldest deadline 2023-06-22T17:27:29.669924+0000) 2023-06-22T17:27:43.173 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:42 smithi133 conmon[107946]: 2023-06-22T17:27:42.959+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:14.172382+0000 front 2023-06-22T17:27:14.172429+0000 (oldest deadline 2023-06-22T17:27:37.072457+0000) 2023-06-22T17:27:43.173 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:42 smithi133 conmon[115049]: 2023-06-22T17:27:42.892+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:43.174 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:42 smithi133 conmon[115049]: 2023-06-22T17:27:42.892+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:43.174 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:42 smithi133 conmon[115049]: 2023-06-22T17:27:42.892+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:43.174 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:42 smithi133 conmon[115049]: 2023-06-22T17:27:42.892+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:13.936567+0000 front 2023-06-22T17:27:13.936510+0000 (oldest deadline 2023-06-22T17:27:36.236444+0000) 2023-06-22T17:27:43.376 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:42 smithi101 conmon[112983]: 2023-06-22T17:27:42.895+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:43.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:42 smithi101 conmon[112983]: 2023-06-22T17:27:42.895+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:43.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:42 smithi101 conmon[112983]: 2023-06-22T17:27:42.895+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:43.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:42 smithi101 conmon[112983]: 2023-06-22T17:27:42.895+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.163707+0000 front 2023-06-22T17:27:04.163629+0000 (oldest deadline 2023-06-22T17:27:30.564301+0000) 2023-06-22T17:27:43.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:42 smithi101 conmon[112983]: 2023-06-22T17:27:42.895+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:14.065051+0000 front 2023-06-22T17:27:14.065033+0000 (oldest deadline 2023-06-22T17:27:39.365200+0000) 2023-06-22T17:27:44.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:43 smithi039 conmon[122618]: 2023-06-22T17:27:43.713+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:44.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:43 smithi039 conmon[122618]: 2023-06-22T17:27:43.713+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:44.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:43 smithi039 conmon[122618]: 2023-06-22T17:27:43.713+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:27:00.343681+0000 front 2023-06-22T17:27:00.343658+0000 (oldest deadline 2023-06-22T17:27:21.443709+0000) 2023-06-22T17:27:44.185 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:43 smithi039 conmon[122618]: 2023-06-22T17:27:43.713+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:06.144593+0000 front 2023-06-22T17:27:06.144629+0000 (oldest deadline 2023-06-22T17:27:31.444781+0000) 2023-06-22T17:27:44.185 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:43 smithi039 conmon[122618]: 2023-06-22T17:27:43.713+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:11.445285+0000 front 2023-06-22T17:27:11.445252+0000 (oldest deadline 2023-06-22T17:27:36.145376+0000) 2023-06-22T17:27:44.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:43 smithi101 conmon[112983]: 2023-06-22T17:27:43.888+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:44.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:43 smithi101 conmon[112983]: 2023-06-22T17:27:43.888+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:44.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:43 smithi101 conmon[112983]: 2023-06-22T17:27:43.888+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:44.378 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:43 smithi101 conmon[112983]: 2023-06-22T17:27:43.888+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.163707+0000 front 2023-06-22T17:27:04.163629+0000 (oldest deadline 2023-06-22T17:27:30.564301+0000) 2023-06-22T17:27:44.378 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:43 smithi101 conmon[112983]: 2023-06-22T17:27:43.888+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:14.065051+0000 front 2023-06-22T17:27:14.065033+0000 (oldest deadline 2023-06-22T17:27:39.365200+0000) 2023-06-22T17:27:44.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:43 smithi133 conmon[115049]: 2023-06-22T17:27:43.928+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:44.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:43 smithi133 conmon[115049]: 2023-06-22T17:27:43.928+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:44.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:43 smithi133 conmon[115049]: 2023-06-22T17:27:43.928+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:44.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:43 smithi133 conmon[115049]: 2023-06-22T17:27:43.928+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:13.936567+0000 front 2023-06-22T17:27:13.936510+0000 (oldest deadline 2023-06-22T17:27:36.236444+0000) 2023-06-22T17:27:44.422 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:43 smithi133 conmon[107946]: 2023-06-22T17:27:43.973+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:44.422 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:43 smithi133 conmon[107946]: 2023-06-22T17:27:43.973+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.767866+0000 front 2023-06-22T17:26:52.767701+0000 (oldest deadline 2023-06-22T17:27:17.467693+0000) 2023-06-22T17:27:44.422 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:43 smithi133 conmon[107946]: 2023-06-22T17:27:43.973+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:59.168703+0000 front 2023-06-22T17:26:59.168748+0000 (oldest deadline 2023-06-22T17:27:21.468879+0000) 2023-06-22T17:27:44.422 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:43 smithi133 conmon[107946]: 2023-06-22T17:27:43.973+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.969905+0000 front 2023-06-22T17:27:04.970073+0000 (oldest deadline 2023-06-22T17:27:29.669924+0000) 2023-06-22T17:27:44.423 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:43 smithi133 conmon[107946]: 2023-06-22T17:27:43.973+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:14.172382+0000 front 2023-06-22T17:27:14.172429+0000 (oldest deadline 2023-06-22T17:27:37.072457+0000) 2023-06-22T17:27:44.423 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:43 smithi133 conmon[107946]: 2023-06-22T17:27:43.973+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:22.873683+0000 front 2023-06-22T17:27:22.873718+0000 (oldest deadline 2023-06-22T17:27:43.973704+0000) 2023-06-22T17:27:45.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:44 smithi133 conmon[115049]: 2023-06-22T17:27:44.917+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:45.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:44 smithi133 conmon[115049]: 2023-06-22T17:27:44.917+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:45.172 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:44 smithi133 conmon[115049]: 2023-06-22T17:27:44.917+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:45.172 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:44 smithi133 conmon[115049]: 2023-06-22T17:27:44.917+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:13.936567+0000 front 2023-06-22T17:27:13.936510+0000 (oldest deadline 2023-06-22T17:27:36.236444+0000) 2023-06-22T17:27:45.172 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:44 smithi133 conmon[115049]: 2023-06-22T17:27:44.917+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:21.438006+0000 front 2023-06-22T17:27:21.438025+0000 (oldest deadline 2023-06-22T17:27:44.338268+0000) 2023-06-22T17:27:45.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:44 smithi133 conmon[107946]: 2023-06-22T17:27:44.930+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:45.172 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:44 smithi133 conmon[107946]: 2023-06-22T17:27:44.930+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.767866+0000 front 2023-06-22T17:26:52.767701+0000 (oldest deadline 2023-06-22T17:27:17.467693+0000) 2023-06-22T17:27:45.173 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:44 smithi133 conmon[107946]: 2023-06-22T17:27:44.930+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:59.168703+0000 front 2023-06-22T17:26:59.168748+0000 (oldest deadline 2023-06-22T17:27:21.468879+0000) 2023-06-22T17:27:45.173 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:44 smithi133 conmon[107946]: 2023-06-22T17:27:44.930+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.969905+0000 front 2023-06-22T17:27:04.970073+0000 (oldest deadline 2023-06-22T17:27:29.669924+0000) 2023-06-22T17:27:45.173 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:44 smithi133 conmon[107946]: 2023-06-22T17:27:44.930+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:14.172382+0000 front 2023-06-22T17:27:14.172429+0000 (oldest deadline 2023-06-22T17:27:37.072457+0000) 2023-06-22T17:27:45.173 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:44 smithi133 conmon[107946]: 2023-06-22T17:27:44.930+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:22.873683+0000 front 2023-06-22T17:27:22.873718+0000 (oldest deadline 2023-06-22T17:27:43.973704+0000) 2023-06-22T17:27:45.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:44 smithi039 conmon[122618]: 2023-06-22T17:27:44.709+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:45.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:44 smithi039 conmon[122618]: 2023-06-22T17:27:44.709+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:45.184 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:44 smithi039 conmon[122618]: 2023-06-22T17:27:44.709+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:27:00.343681+0000 front 2023-06-22T17:27:00.343658+0000 (oldest deadline 2023-06-22T17:27:21.443709+0000) 2023-06-22T17:27:45.185 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:44 smithi039 conmon[122618]: 2023-06-22T17:27:44.709+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:06.144593+0000 front 2023-06-22T17:27:06.144629+0000 (oldest deadline 2023-06-22T17:27:31.444781+0000) 2023-06-22T17:27:45.185 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:44 smithi039 conmon[122618]: 2023-06-22T17:27:44.709+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:11.445285+0000 front 2023-06-22T17:27:11.445252+0000 (oldest deadline 2023-06-22T17:27:36.145376+0000) 2023-06-22T17:27:45.185 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:44 smithi039 conmon[122618]: 2023-06-22T17:27:44.709+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:20.547886+0000 front 2023-06-22T17:27:20.547899+0000 (oldest deadline 2023-06-22T17:27:43.948571+0000) 2023-06-22T17:27:45.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:44 smithi101 conmon[112983]: 2023-06-22T17:27:44.893+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:45.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:44 smithi101 conmon[112983]: 2023-06-22T17:27:44.893+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:45.378 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:44 smithi101 conmon[112983]: 2023-06-22T17:27:44.893+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:45.378 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:44 smithi101 conmon[112983]: 2023-06-22T17:27:44.893+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.163707+0000 front 2023-06-22T17:27:04.163629+0000 (oldest deadline 2023-06-22T17:27:30.564301+0000) 2023-06-22T17:27:45.378 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:44 smithi101 conmon[112983]: 2023-06-22T17:27:44.893+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:14.065051+0000 front 2023-06-22T17:27:14.065033+0000 (oldest deadline 2023-06-22T17:27:39.365200+0000) 2023-06-22T17:27:45.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:45 smithi039 conmon[122618]: 2023-06-22T17:27:45.677+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:45.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:45 smithi039 conmon[122618]: 2023-06-22T17:27:45.677+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:45.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:45 smithi039 conmon[122618]: 2023-06-22T17:27:45.677+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:27:00.343681+0000 front 2023-06-22T17:27:00.343658+0000 (oldest deadline 2023-06-22T17:27:21.443709+0000) 2023-06-22T17:27:45.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:45 smithi039 conmon[122618]: 2023-06-22T17:27:45.677+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:06.144593+0000 front 2023-06-22T17:27:06.144629+0000 (oldest deadline 2023-06-22T17:27:31.444781+0000) 2023-06-22T17:27:45.935 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:45 smithi039 conmon[122618]: 2023-06-22T17:27:45.677+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:11.445285+0000 front 2023-06-22T17:27:11.445252+0000 (oldest deadline 2023-06-22T17:27:36.145376+0000) 2023-06-22T17:27:45.935 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:45 smithi039 conmon[122618]: 2023-06-22T17:27:45.677+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:20.547886+0000 front 2023-06-22T17:27:20.547899+0000 (oldest deadline 2023-06-22T17:27:43.948571+0000) 2023-06-22T17:27:46.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:45 smithi101 conmon[112983]: 2023-06-22T17:27:45.921+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:46.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:45 smithi101 conmon[112983]: 2023-06-22T17:27:45.921+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:46.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:45 smithi101 conmon[112983]: 2023-06-22T17:27:45.921+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:46.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:45 smithi101 conmon[112983]: 2023-06-22T17:27:45.921+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.163707+0000 front 2023-06-22T17:27:04.163629+0000 (oldest deadline 2023-06-22T17:27:30.564301+0000) 2023-06-22T17:27:46.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:45 smithi101 conmon[112983]: 2023-06-22T17:27:45.921+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:14.065051+0000 front 2023-06-22T17:27:14.065033+0000 (oldest deadline 2023-06-22T17:27:39.365200+0000) 2023-06-22T17:27:46.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:45 smithi101 conmon[112983]: 2023-06-22T17:27:45.921+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:19.365904+0000 front 2023-06-22T17:27:19.365888+0000 (oldest deadline 2023-06-22T17:27:45.265802+0000) 2023-06-22T17:27:46.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:45 smithi133 conmon[115049]: 2023-06-22T17:27:45.966+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:46.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:45 smithi133 conmon[115049]: 2023-06-22T17:27:45.966+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:46.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:45 smithi133 conmon[115049]: 2023-06-22T17:27:45.966+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:46.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:45 smithi133 conmon[115049]: 2023-06-22T17:27:45.966+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:13.936567+0000 front 2023-06-22T17:27:13.936510+0000 (oldest deadline 2023-06-22T17:27:36.236444+0000) 2023-06-22T17:27:46.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:45 smithi133 conmon[115049]: 2023-06-22T17:27:45.966+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:21.438006+0000 front 2023-06-22T17:27:21.438025+0000 (oldest deadline 2023-06-22T17:27:44.338268+0000) 2023-06-22T17:27:46.422 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:45 smithi133 conmon[107946]: 2023-06-22T17:27:45.945+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:42.866167+0000 front 2023-06-22T17:26:42.865974+0000 (oldest deadline 2023-06-22T17:27:06.966106+0000) 2023-06-22T17:27:46.422 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:45 smithi133 conmon[107946]: 2023-06-22T17:27:45.945+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:52.767866+0000 front 2023-06-22T17:26:52.767701+0000 (oldest deadline 2023-06-22T17:27:17.467693+0000) 2023-06-22T17:27:46.422 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:45 smithi133 conmon[107946]: 2023-06-22T17:27:45.945+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:59.168703+0000 front 2023-06-22T17:26:59.168748+0000 (oldest deadline 2023-06-22T17:27:21.468879+0000) 2023-06-22T17:27:46.423 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:45 smithi133 conmon[107946]: 2023-06-22T17:27:45.945+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.969905+0000 front 2023-06-22T17:27:04.970073+0000 (oldest deadline 2023-06-22T17:27:29.669924+0000) 2023-06-22T17:27:46.423 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:45 smithi133 conmon[107946]: 2023-06-22T17:27:45.945+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:14.172382+0000 front 2023-06-22T17:27:14.172429+0000 (oldest deadline 2023-06-22T17:27:37.072457+0000) 2023-06-22T17:27:46.423 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:45 smithi133 conmon[107946]: 2023-06-22T17:27:45.945+0000 7f7176bbc700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:22.873683+0000 front 2023-06-22T17:27:22.873718+0000 (oldest deadline 2023-06-22T17:27:43.973704+0000) 2023-06-22T17:27:46.933 INFO:journalctl@ceph.osd.8.smithi133.stdout:Jun 22 17:27:46 smithi133 bash[124652]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c-osd-8 2023-06-22T17:27:46.933 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:46 smithi039 conmon[122618]: 2023-06-22T17:27:46.633+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:46.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:46 smithi039 conmon[122618]: 2023-06-22T17:27:46.633+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:46.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:46 smithi039 conmon[122618]: 2023-06-22T17:27:46.633+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:27:00.343681+0000 front 2023-06-22T17:27:00.343658+0000 (oldest deadline 2023-06-22T17:27:21.443709+0000) 2023-06-22T17:27:46.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:46 smithi039 conmon[122618]: 2023-06-22T17:27:46.633+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:06.144593+0000 front 2023-06-22T17:27:06.144629+0000 (oldest deadline 2023-06-22T17:27:31.444781+0000) 2023-06-22T17:27:46.935 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:46 smithi039 conmon[122618]: 2023-06-22T17:27:46.633+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:11.445285+0000 front 2023-06-22T17:27:11.445252+0000 (oldest deadline 2023-06-22T17:27:36.145376+0000) 2023-06-22T17:27:46.935 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:46 smithi039 conmon[122618]: 2023-06-22T17:27:46.633+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:20.547886+0000 front 2023-06-22T17:27:20.547899+0000 (oldest deadline 2023-06-22T17:27:43.948571+0000) 2023-06-22T17:27:47.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:46 smithi101 conmon[112983]: 2023-06-22T17:27:46.915+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:47.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:46 smithi101 conmon[112983]: 2023-06-22T17:27:46.915+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:47.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:46 smithi101 conmon[112983]: 2023-06-22T17:27:46.915+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:47.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:46 smithi101 conmon[112983]: 2023-06-22T17:27:46.915+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.163707+0000 front 2023-06-22T17:27:04.163629+0000 (oldest deadline 2023-06-22T17:27:30.564301+0000) 2023-06-22T17:27:47.378 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:46 smithi101 conmon[112983]: 2023-06-22T17:27:46.915+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:14.065051+0000 front 2023-06-22T17:27:14.065033+0000 (oldest deadline 2023-06-22T17:27:39.365200+0000) 2023-06-22T17:27:47.378 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:46 smithi101 conmon[112983]: 2023-06-22T17:27:46.915+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:19.365904+0000 front 2023-06-22T17:27:19.365888+0000 (oldest deadline 2023-06-22T17:27:45.265802+0000) 2023-06-22T17:27:47.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:46 smithi133 conmon[115049]: 2023-06-22T17:27:46.929+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:47.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:46 smithi133 conmon[115049]: 2023-06-22T17:27:46.929+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:47.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:46 smithi133 conmon[115049]: 2023-06-22T17:27:46.929+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:47.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:46 smithi133 conmon[115049]: 2023-06-22T17:27:46.929+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:13.936567+0000 front 2023-06-22T17:27:13.936510+0000 (oldest deadline 2023-06-22T17:27:36.236444+0000) 2023-06-22T17:27:47.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:46 smithi133 conmon[115049]: 2023-06-22T17:27:46.929+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:21.438006+0000 front 2023-06-22T17:27:21.438025+0000 (oldest deadline 2023-06-22T17:27:44.338268+0000) 2023-06-22T17:27:47.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:47 smithi039 conmon[122618]: 2023-06-22T17:27:47.624+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:47.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:47 smithi039 conmon[122618]: 2023-06-22T17:27:47.624+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:47.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:47 smithi039 conmon[122618]: 2023-06-22T17:27:47.624+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:27:00.343681+0000 front 2023-06-22T17:27:00.343658+0000 (oldest deadline 2023-06-22T17:27:21.443709+0000) 2023-06-22T17:27:47.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:47 smithi039 conmon[122618]: 2023-06-22T17:27:47.624+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:06.144593+0000 front 2023-06-22T17:27:06.144629+0000 (oldest deadline 2023-06-22T17:27:31.444781+0000) 2023-06-22T17:27:47.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:47 smithi039 conmon[122618]: 2023-06-22T17:27:47.624+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:11.445285+0000 front 2023-06-22T17:27:11.445252+0000 (oldest deadline 2023-06-22T17:27:36.145376+0000) 2023-06-22T17:27:47.935 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:47 smithi039 conmon[122618]: 2023-06-22T17:27:47.624+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:20.547886+0000 front 2023-06-22T17:27:20.547899+0000 (oldest deadline 2023-06-22T17:27:43.948571+0000) 2023-06-22T17:27:48.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:47 smithi133 conmon[115049]: 2023-06-22T17:27:47.963+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:48.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:47 smithi133 conmon[115049]: 2023-06-22T17:27:47.963+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:48.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:47 smithi133 conmon[115049]: 2023-06-22T17:27:47.963+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:48.172 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:47 smithi133 conmon[115049]: 2023-06-22T17:27:47.963+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:13.936567+0000 front 2023-06-22T17:27:13.936510+0000 (oldest deadline 2023-06-22T17:27:36.236444+0000) 2023-06-22T17:27:48.172 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:47 smithi133 conmon[115049]: 2023-06-22T17:27:47.963+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:21.438006+0000 front 2023-06-22T17:27:21.438025+0000 (oldest deadline 2023-06-22T17:27:44.338268+0000) 2023-06-22T17:27:48.299 DEBUG:teuthology.orchestra.run.smithi133:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.8.service' 2023-06-22T17:27:48.338 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-22T17:27:48.339 INFO:tasks.cephadm.osd.8:Stopped osd.8 2023-06-22T17:27:48.340 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2023-06-22T17:27:48.340 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.9 2023-06-22T17:27:48.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:47 smithi101 conmon[112983]: 2023-06-22T17:27:47.878+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:48.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:47 smithi101 conmon[112983]: 2023-06-22T17:27:47.878+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:48.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:47 smithi101 conmon[112983]: 2023-06-22T17:27:47.878+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:48.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:47 smithi101 conmon[112983]: 2023-06-22T17:27:47.878+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.163707+0000 front 2023-06-22T17:27:04.163629+0000 (oldest deadline 2023-06-22T17:27:30.564301+0000) 2023-06-22T17:27:48.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:47 smithi101 conmon[112983]: 2023-06-22T17:27:47.878+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:14.065051+0000 front 2023-06-22T17:27:14.065033+0000 (oldest deadline 2023-06-22T17:27:39.365200+0000) 2023-06-22T17:27:48.378 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:47 smithi101 conmon[112983]: 2023-06-22T17:27:47.878+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:19.365904+0000 front 2023-06-22T17:27:19.365888+0000 (oldest deadline 2023-06-22T17:27:45.265802+0000) 2023-06-22T17:27:48.624 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:48 smithi039 systemd[1]: Stopping Ceph osd.9 for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:27:48.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:48 smithi039 conmon[122618]: 2023-06-22T17:27:48.621+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:48.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:48 smithi039 conmon[122618]: 2023-06-22T17:27:48.621+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:48.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:48 smithi039 conmon[122618]: 2023-06-22T17:27:48.621+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:27:00.343681+0000 front 2023-06-22T17:27:00.343658+0000 (oldest deadline 2023-06-22T17:27:21.443709+0000) 2023-06-22T17:27:48.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:48 smithi039 conmon[122618]: 2023-06-22T17:27:48.621+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:06.144593+0000 front 2023-06-22T17:27:06.144629+0000 (oldest deadline 2023-06-22T17:27:31.444781+0000) 2023-06-22T17:27:48.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:48 smithi039 conmon[122618]: 2023-06-22T17:27:48.621+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:11.445285+0000 front 2023-06-22T17:27:11.445252+0000 (oldest deadline 2023-06-22T17:27:36.145376+0000) 2023-06-22T17:27:48.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:48 smithi039 conmon[122618]: 2023-06-22T17:27:48.621+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:20.547886+0000 front 2023-06-22T17:27:20.547899+0000 (oldest deadline 2023-06-22T17:27:43.948571+0000) 2023-06-22T17:27:48.935 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:48 smithi039 conmon[122618]: 2023-06-22T17:27:48.693+0000 7f7e1149e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-22T17:27:48.935 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:48 smithi039 conmon[122618]: 2023-06-22T17:27:48.693+0000 7f7e1149e700 -1 osd.9 81 *** Got signal Terminated *** 2023-06-22T17:27:48.935 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:48 smithi039 conmon[122618]: 2023-06-22T17:27:48.693+0000 7f7e1149e700 -1 osd.9 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-06-22T17:27:49.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:48 smithi101 conmon[112983]: 2023-06-22T17:27:48.829+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:49.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:48 smithi101 conmon[112983]: 2023-06-22T17:27:48.829+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:49.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:48 smithi101 conmon[112983]: 2023-06-22T17:27:48.829+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:49.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:48 smithi101 conmon[112983]: 2023-06-22T17:27:48.830+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.163707+0000 front 2023-06-22T17:27:04.163629+0000 (oldest deadline 2023-06-22T17:27:30.564301+0000) 2023-06-22T17:27:49.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:48 smithi101 conmon[112983]: 2023-06-22T17:27:48.830+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:14.065051+0000 front 2023-06-22T17:27:14.065033+0000 (oldest deadline 2023-06-22T17:27:39.365200+0000) 2023-06-22T17:27:49.128 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:48 smithi101 conmon[112983]: 2023-06-22T17:27:48.830+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:19.365904+0000 front 2023-06-22T17:27:19.365888+0000 (oldest deadline 2023-06-22T17:27:45.265802+0000) 2023-06-22T17:27:49.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:48 smithi133 conmon[115049]: 2023-06-22T17:27:48.983+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:49.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:48 smithi133 conmon[115049]: 2023-06-22T17:27:48.983+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:49.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:48 smithi133 conmon[115049]: 2023-06-22T17:27:48.983+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:49.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:48 smithi133 conmon[115049]: 2023-06-22T17:27:48.983+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:13.936567+0000 front 2023-06-22T17:27:13.936510+0000 (oldest deadline 2023-06-22T17:27:36.236444+0000) 2023-06-22T17:27:49.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:48 smithi133 conmon[115049]: 2023-06-22T17:27:48.983+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:21.438006+0000 front 2023-06-22T17:27:21.438025+0000 (oldest deadline 2023-06-22T17:27:44.338268+0000) 2023-06-22T17:27:49.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:49 smithi039 conmon[122618]: 2023-06-22T17:27:49.663+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:49.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:49 smithi039 conmon[122618]: 2023-06-22T17:27:49.663+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:49.935 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:49 smithi039 conmon[122618]: 2023-06-22T17:27:49.663+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:27:00.343681+0000 front 2023-06-22T17:27:00.343658+0000 (oldest deadline 2023-06-22T17:27:21.443709+0000) 2023-06-22T17:27:49.935 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:49 smithi039 conmon[122618]: 2023-06-22T17:27:49.663+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:06.144593+0000 front 2023-06-22T17:27:06.144629+0000 (oldest deadline 2023-06-22T17:27:31.444781+0000) 2023-06-22T17:27:49.935 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:49 smithi039 conmon[122618]: 2023-06-22T17:27:49.663+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:11.445285+0000 front 2023-06-22T17:27:11.445252+0000 (oldest deadline 2023-06-22T17:27:36.145376+0000) 2023-06-22T17:27:49.935 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:49 smithi039 conmon[122618]: 2023-06-22T17:27:49.663+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:20.547886+0000 front 2023-06-22T17:27:20.547899+0000 (oldest deadline 2023-06-22T17:27:43.948571+0000) 2023-06-22T17:27:50.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:49 smithi101 conmon[112983]: 2023-06-22T17:27:49.878+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:50.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:49 smithi101 conmon[112983]: 2023-06-22T17:27:49.878+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:50.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:49 smithi101 conmon[112983]: 2023-06-22T17:27:49.878+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:50.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:49 smithi101 conmon[112983]: 2023-06-22T17:27:49.878+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.163707+0000 front 2023-06-22T17:27:04.163629+0000 (oldest deadline 2023-06-22T17:27:30.564301+0000) 2023-06-22T17:27:50.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:49 smithi101 conmon[112983]: 2023-06-22T17:27:49.878+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:14.065051+0000 front 2023-06-22T17:27:14.065033+0000 (oldest deadline 2023-06-22T17:27:39.365200+0000) 2023-06-22T17:27:50.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:49 smithi101 conmon[112983]: 2023-06-22T17:27:49.878+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:19.365904+0000 front 2023-06-22T17:27:19.365888+0000 (oldest deadline 2023-06-22T17:27:45.265802+0000) 2023-06-22T17:27:50.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:49 smithi133 conmon[115049]: 2023-06-22T17:27:49.967+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:50.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:49 smithi133 conmon[115049]: 2023-06-22T17:27:49.967+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:50.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:49 smithi133 conmon[115049]: 2023-06-22T17:27:49.967+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:50.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:49 smithi133 conmon[115049]: 2023-06-22T17:27:49.967+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:13.936567+0000 front 2023-06-22T17:27:13.936510+0000 (oldest deadline 2023-06-22T17:27:36.236444+0000) 2023-06-22T17:27:50.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:49 smithi133 conmon[115049]: 2023-06-22T17:27:49.967+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:21.438006+0000 front 2023-06-22T17:27:21.438025+0000 (oldest deadline 2023-06-22T17:27:44.338268+0000) 2023-06-22T17:27:50.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:50 smithi039 conmon[122618]: 2023-06-22T17:27:50.630+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:50.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:50 smithi039 conmon[122618]: 2023-06-22T17:27:50.630+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:50.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:50 smithi039 conmon[122618]: 2023-06-22T17:27:50.630+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:27:00.343681+0000 front 2023-06-22T17:27:00.343658+0000 (oldest deadline 2023-06-22T17:27:21.443709+0000) 2023-06-22T17:27:50.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:50 smithi039 conmon[122618]: 2023-06-22T17:27:50.630+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:06.144593+0000 front 2023-06-22T17:27:06.144629+0000 (oldest deadline 2023-06-22T17:27:31.444781+0000) 2023-06-22T17:27:50.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:50 smithi039 conmon[122618]: 2023-06-22T17:27:50.630+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:11.445285+0000 front 2023-06-22T17:27:11.445252+0000 (oldest deadline 2023-06-22T17:27:36.145376+0000) 2023-06-22T17:27:50.935 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:50 smithi039 conmon[122618]: 2023-06-22T17:27:50.630+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:20.547886+0000 front 2023-06-22T17:27:20.547899+0000 (oldest deadline 2023-06-22T17:27:43.948571+0000) 2023-06-22T17:27:51.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:50 smithi101 conmon[112983]: 2023-06-22T17:27:50.900+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:51.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:50 smithi101 conmon[112983]: 2023-06-22T17:27:50.900+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:51.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:50 smithi101 conmon[112983]: 2023-06-22T17:27:50.900+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:51.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:50 smithi101 conmon[112983]: 2023-06-22T17:27:50.900+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.163707+0000 front 2023-06-22T17:27:04.163629+0000 (oldest deadline 2023-06-22T17:27:30.564301+0000) 2023-06-22T17:27:51.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:50 smithi101 conmon[112983]: 2023-06-22T17:27:50.900+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:14.065051+0000 front 2023-06-22T17:27:14.065033+0000 (oldest deadline 2023-06-22T17:27:39.365200+0000) 2023-06-22T17:27:51.378 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:50 smithi101 conmon[112983]: 2023-06-22T17:27:50.900+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:19.365904+0000 front 2023-06-22T17:27:19.365888+0000 (oldest deadline 2023-06-22T17:27:45.265802+0000) 2023-06-22T17:27:51.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:50 smithi133 conmon[115049]: 2023-06-22T17:27:50.971+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:51.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:50 smithi133 conmon[115049]: 2023-06-22T17:27:50.971+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:51.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:50 smithi133 conmon[115049]: 2023-06-22T17:27:50.971+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:51.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:50 smithi133 conmon[115049]: 2023-06-22T17:27:50.971+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:13.936567+0000 front 2023-06-22T17:27:13.936510+0000 (oldest deadline 2023-06-22T17:27:36.236444+0000) 2023-06-22T17:27:51.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:50 smithi133 conmon[115049]: 2023-06-22T17:27:50.971+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:21.438006+0000 front 2023-06-22T17:27:21.438025+0000 (oldest deadline 2023-06-22T17:27:44.338268+0000) 2023-06-22T17:27:51.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:51 smithi039 conmon[122618]: 2023-06-22T17:27:51.608+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:51.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:51 smithi039 conmon[122618]: 2023-06-22T17:27:51.608+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:51.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:51 smithi039 conmon[122618]: 2023-06-22T17:27:51.608+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:27:00.343681+0000 front 2023-06-22T17:27:00.343658+0000 (oldest deadline 2023-06-22T17:27:21.443709+0000) 2023-06-22T17:27:51.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:51 smithi039 conmon[122618]: 2023-06-22T17:27:51.608+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:06.144593+0000 front 2023-06-22T17:27:06.144629+0000 (oldest deadline 2023-06-22T17:27:31.444781+0000) 2023-06-22T17:27:51.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:51 smithi039 conmon[122618]: 2023-06-22T17:27:51.608+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:11.445285+0000 front 2023-06-22T17:27:11.445252+0000 (oldest deadline 2023-06-22T17:27:36.145376+0000) 2023-06-22T17:27:51.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:51 smithi039 conmon[122618]: 2023-06-22T17:27:51.608+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:20.547886+0000 front 2023-06-22T17:27:20.547899+0000 (oldest deadline 2023-06-22T17:27:43.948571+0000) 2023-06-22T17:27:52.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:51 smithi101 conmon[112983]: 2023-06-22T17:27:51.863+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:52.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:51 smithi101 conmon[112983]: 2023-06-22T17:27:51.863+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:52.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:51 smithi101 conmon[112983]: 2023-06-22T17:27:51.863+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:52.128 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:51 smithi101 conmon[112983]: 2023-06-22T17:27:51.863+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.163707+0000 front 2023-06-22T17:27:04.163629+0000 (oldest deadline 2023-06-22T17:27:30.564301+0000) 2023-06-22T17:27:52.128 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:51 smithi101 conmon[112983]: 2023-06-22T17:27:51.863+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:14.065051+0000 front 2023-06-22T17:27:14.065033+0000 (oldest deadline 2023-06-22T17:27:39.365200+0000) 2023-06-22T17:27:52.128 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:51 smithi101 conmon[112983]: 2023-06-22T17:27:51.863+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:19.365904+0000 front 2023-06-22T17:27:19.365888+0000 (oldest deadline 2023-06-22T17:27:45.265802+0000) 2023-06-22T17:27:52.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:51 smithi133 conmon[115049]: 2023-06-22T17:27:51.943+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:52.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:51 smithi133 conmon[115049]: 2023-06-22T17:27:51.943+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:52.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:51 smithi133 conmon[115049]: 2023-06-22T17:27:51.943+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:52.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:51 smithi133 conmon[115049]: 2023-06-22T17:27:51.943+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:13.936567+0000 front 2023-06-22T17:27:13.936510+0000 (oldest deadline 2023-06-22T17:27:36.236444+0000) 2023-06-22T17:27:52.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:51 smithi133 conmon[115049]: 2023-06-22T17:27:51.943+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:21.438006+0000 front 2023-06-22T17:27:21.438025+0000 (oldest deadline 2023-06-22T17:27:44.338268+0000) 2023-06-22T17:27:52.423 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:51 smithi133 conmon[115049]: 2023-06-22T17:27:51.943+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2023-06-22T17:27:29.039130+0000 front 2023-06-22T17:27:29.039276+0000 (oldest deadline 2023-06-22T17:27:51.939189+0000) 2023-06-22T17:27:52.933 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:52 smithi039 conmon[122618]: 2023-06-22T17:27:52.621+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:52.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:52 smithi039 conmon[122618]: 2023-06-22T17:27:52.621+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:52.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:52 smithi039 conmon[122618]: 2023-06-22T17:27:52.621+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:27:00.343681+0000 front 2023-06-22T17:27:00.343658+0000 (oldest deadline 2023-06-22T17:27:21.443709+0000) 2023-06-22T17:27:52.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:52 smithi039 conmon[122618]: 2023-06-22T17:27:52.621+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:06.144593+0000 front 2023-06-22T17:27:06.144629+0000 (oldest deadline 2023-06-22T17:27:31.444781+0000) 2023-06-22T17:27:52.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:52 smithi039 conmon[122618]: 2023-06-22T17:27:52.621+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:11.445285+0000 front 2023-06-22T17:27:11.445252+0000 (oldest deadline 2023-06-22T17:27:36.145376+0000) 2023-06-22T17:27:52.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:52 smithi039 conmon[122618]: 2023-06-22T17:27:52.621+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:20.547886+0000 front 2023-06-22T17:27:20.547899+0000 (oldest deadline 2023-06-22T17:27:43.948571+0000) 2023-06-22T17:27:53.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:52 smithi101 conmon[112983]: 2023-06-22T17:27:52.862+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:53.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:52 smithi101 conmon[112983]: 2023-06-22T17:27:52.862+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:53.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:52 smithi101 conmon[112983]: 2023-06-22T17:27:52.862+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:53.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:52 smithi101 conmon[112983]: 2023-06-22T17:27:52.862+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.163707+0000 front 2023-06-22T17:27:04.163629+0000 (oldest deadline 2023-06-22T17:27:30.564301+0000) 2023-06-22T17:27:53.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:52 smithi101 conmon[112983]: 2023-06-22T17:27:52.862+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:14.065051+0000 front 2023-06-22T17:27:14.065033+0000 (oldest deadline 2023-06-22T17:27:39.365200+0000) 2023-06-22T17:27:53.128 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:52 smithi101 conmon[112983]: 2023-06-22T17:27:52.862+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:19.365904+0000 front 2023-06-22T17:27:19.365888+0000 (oldest deadline 2023-06-22T17:27:45.265802+0000) 2023-06-22T17:27:53.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:52 smithi133 conmon[115049]: 2023-06-22T17:27:52.956+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:53.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:52 smithi133 conmon[115049]: 2023-06-22T17:27:52.956+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:53.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:52 smithi133 conmon[115049]: 2023-06-22T17:27:52.956+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:53.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:52 smithi133 conmon[115049]: 2023-06-22T17:27:52.956+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:13.936567+0000 front 2023-06-22T17:27:13.936510+0000 (oldest deadline 2023-06-22T17:27:36.236444+0000) 2023-06-22T17:27:53.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:52 smithi133 conmon[115049]: 2023-06-22T17:27:52.956+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:21.438006+0000 front 2023-06-22T17:27:21.438025+0000 (oldest deadline 2023-06-22T17:27:44.338268+0000) 2023-06-22T17:27:53.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:52 smithi133 conmon[115049]: 2023-06-22T17:27:52.956+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2023-06-22T17:27:29.039130+0000 front 2023-06-22T17:27:29.039276+0000 (oldest deadline 2023-06-22T17:27:51.939189+0000) 2023-06-22T17:27:53.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:53 smithi039 conmon[122618]: 2023-06-22T17:27:53.607+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:39.940323+0000 front 2023-06-22T17:26:39.940348+0000 (oldest deadline 2023-06-22T17:27:05.840483+0000) 2023-06-22T17:27:53.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:53 smithi039 conmon[122618]: 2023-06-22T17:27:53.607+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.341442+0000 front 2023-06-22T17:26:49.341548+0000 (oldest deadline 2023-06-22T17:27:15.241604+0000) 2023-06-22T17:27:53.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:53 smithi039 conmon[122618]: 2023-06-22T17:27:53.607+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:27:00.343681+0000 front 2023-06-22T17:27:00.343658+0000 (oldest deadline 2023-06-22T17:27:21.443709+0000) 2023-06-22T17:27:53.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:53 smithi039 conmon[122618]: 2023-06-22T17:27:53.607+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:06.144593+0000 front 2023-06-22T17:27:06.144629+0000 (oldest deadline 2023-06-22T17:27:31.444781+0000) 2023-06-22T17:27:53.934 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:53 smithi039 conmon[122618]: 2023-06-22T17:27:53.607+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:11.445285+0000 front 2023-06-22T17:27:11.445252+0000 (oldest deadline 2023-06-22T17:27:36.145376+0000) 2023-06-22T17:27:53.935 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:53 smithi039 conmon[122618]: 2023-06-22T17:27:53.607+0000 7f7e0ca9a700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:20.547886+0000 front 2023-06-22T17:27:20.547899+0000 (oldest deadline 2023-06-22T17:27:43.948571+0000) 2023-06-22T17:27:54.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:53 smithi101 conmon[112983]: 2023-06-22T17:27:53.889+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:54.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:53 smithi101 conmon[112983]: 2023-06-22T17:27:53.889+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:54.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:53 smithi101 conmon[112983]: 2023-06-22T17:27:53.889+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:54.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:53 smithi101 conmon[112983]: 2023-06-22T17:27:53.889+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.163707+0000 front 2023-06-22T17:27:04.163629+0000 (oldest deadline 2023-06-22T17:27:30.564301+0000) 2023-06-22T17:27:54.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:53 smithi101 conmon[112983]: 2023-06-22T17:27:53.889+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:14.065051+0000 front 2023-06-22T17:27:14.065033+0000 (oldest deadline 2023-06-22T17:27:39.365200+0000) 2023-06-22T17:27:54.378 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:53 smithi101 conmon[112983]: 2023-06-22T17:27:53.889+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:19.365904+0000 front 2023-06-22T17:27:19.365888+0000 (oldest deadline 2023-06-22T17:27:45.265802+0000) 2023-06-22T17:27:54.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:54 smithi133 conmon[115049]: 2023-06-22T17:27:54.007+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:54.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:54 smithi133 conmon[115049]: 2023-06-22T17:27:54.007+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:54.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:54 smithi133 conmon[115049]: 2023-06-22T17:27:54.007+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:54.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:54 smithi133 conmon[115049]: 2023-06-22T17:27:54.007+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:13.936567+0000 front 2023-06-22T17:27:13.936510+0000 (oldest deadline 2023-06-22T17:27:36.236444+0000) 2023-06-22T17:27:54.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:54 smithi133 conmon[115049]: 2023-06-22T17:27:54.007+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:21.438006+0000 front 2023-06-22T17:27:21.438025+0000 (oldest deadline 2023-06-22T17:27:44.338268+0000) 2023-06-22T17:27:54.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:54 smithi133 conmon[115049]: 2023-06-22T17:27:54.007+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2023-06-22T17:27:29.039130+0000 front 2023-06-22T17:27:29.039276+0000 (oldest deadline 2023-06-22T17:27:51.939189+0000) 2023-06-22T17:27:54.684 INFO:journalctl@ceph.osd.9.smithi039.stdout:Jun 22 17:27:54 smithi039 bash[135165]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c-osd-9 2023-06-22T17:27:55.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:54 smithi101 conmon[112983]: 2023-06-22T17:27:54.894+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:55.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:54 smithi101 conmon[112983]: 2023-06-22T17:27:54.894+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:55.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:54 smithi101 conmon[112983]: 2023-06-22T17:27:54.894+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:55.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:54 smithi101 conmon[112983]: 2023-06-22T17:27:54.894+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.163707+0000 front 2023-06-22T17:27:04.163629+0000 (oldest deadline 2023-06-22T17:27:30.564301+0000) 2023-06-22T17:27:55.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:54 smithi101 conmon[112983]: 2023-06-22T17:27:54.894+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:14.065051+0000 front 2023-06-22T17:27:14.065033+0000 (oldest deadline 2023-06-22T17:27:39.365200+0000) 2023-06-22T17:27:55.378 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:54 smithi101 conmon[112983]: 2023-06-22T17:27:54.894+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:19.365904+0000 front 2023-06-22T17:27:19.365888+0000 (oldest deadline 2023-06-22T17:27:45.265802+0000) 2023-06-22T17:27:55.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:54 smithi133 conmon[115049]: 2023-06-22T17:27:54.960+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:55.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:54 smithi133 conmon[115049]: 2023-06-22T17:27:54.960+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:55.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:54 smithi133 conmon[115049]: 2023-06-22T17:27:54.960+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:55.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:54 smithi133 conmon[115049]: 2023-06-22T17:27:54.960+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:13.936567+0000 front 2023-06-22T17:27:13.936510+0000 (oldest deadline 2023-06-22T17:27:36.236444+0000) 2023-06-22T17:27:55.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:54 smithi133 conmon[115049]: 2023-06-22T17:27:54.960+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:21.438006+0000 front 2023-06-22T17:27:21.438025+0000 (oldest deadline 2023-06-22T17:27:44.338268+0000) 2023-06-22T17:27:55.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:54 smithi133 conmon[115049]: 2023-06-22T17:27:54.960+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2023-06-22T17:27:29.039130+0000 front 2023-06-22T17:27:29.039276+0000 (oldest deadline 2023-06-22T17:27:51.939189+0000) 2023-06-22T17:27:55.971 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.9.service' 2023-06-22T17:27:56.009 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-22T17:27:56.010 INFO:tasks.cephadm.osd.9:Stopped osd.9 2023-06-22T17:27:56.010 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2023-06-22T17:27:56.011 DEBUG:teuthology.orchestra.run.smithi101:> sudo systemctl stop ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.10 2023-06-22T17:27:56.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:55 smithi101 conmon[112983]: 2023-06-22T17:27:55.870+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:56.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:55 smithi101 conmon[112983]: 2023-06-22T17:27:55.870+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:56.127 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:55 smithi101 conmon[112983]: 2023-06-22T17:27:55.870+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:56.128 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:55 smithi101 conmon[112983]: 2023-06-22T17:27:55.870+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.163707+0000 front 2023-06-22T17:27:04.163629+0000 (oldest deadline 2023-06-22T17:27:30.564301+0000) 2023-06-22T17:27:56.128 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:55 smithi101 conmon[112983]: 2023-06-22T17:27:55.870+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:14.065051+0000 front 2023-06-22T17:27:14.065033+0000 (oldest deadline 2023-06-22T17:27:39.365200+0000) 2023-06-22T17:27:56.128 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:55 smithi101 conmon[112983]: 2023-06-22T17:27:55.870+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:19.365904+0000 front 2023-06-22T17:27:19.365888+0000 (oldest deadline 2023-06-22T17:27:45.265802+0000) 2023-06-22T17:27:56.128 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:55 smithi101 conmon[112983]: 2023-06-22T17:27:55.870+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2023-06-22T17:27:29.867207+0000 front 2023-06-22T17:27:29.867162+0000 (oldest deadline 2023-06-22T17:27:55.766998+0000) 2023-06-22T17:27:56.128 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:56 smithi101 systemd[1]: Stopping Ceph osd.10 for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:27:56.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:55 smithi133 conmon[115049]: 2023-06-22T17:27:55.917+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:56.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:55 smithi133 conmon[115049]: 2023-06-22T17:27:55.917+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:56.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:55 smithi133 conmon[115049]: 2023-06-22T17:27:55.917+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:56.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:55 smithi133 conmon[115049]: 2023-06-22T17:27:55.917+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:13.936567+0000 front 2023-06-22T17:27:13.936510+0000 (oldest deadline 2023-06-22T17:27:36.236444+0000) 2023-06-22T17:27:56.172 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:55 smithi133 conmon[115049]: 2023-06-22T17:27:55.917+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:21.438006+0000 front 2023-06-22T17:27:21.438025+0000 (oldest deadline 2023-06-22T17:27:44.338268+0000) 2023-06-22T17:27:56.172 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:55 smithi133 conmon[115049]: 2023-06-22T17:27:55.917+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2023-06-22T17:27:29.039130+0000 front 2023-06-22T17:27:29.039276+0000 (oldest deadline 2023-06-22T17:27:51.939189+0000) 2023-06-22T17:27:56.626 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:56 smithi101 conmon[112983]: 2023-06-22T17:27:56.363+0000 7fe925de1700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-22T17:27:56.627 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:56 smithi101 conmon[112983]: 2023-06-22T17:27:56.363+0000 7fe925de1700 -1 osd.10 81 *** Got signal Terminated *** 2023-06-22T17:27:56.627 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:56 smithi101 conmon[112983]: 2023-06-22T17:27:56.363+0000 7fe925de1700 -1 osd.10 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-06-22T17:27:57.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:56 smithi101 conmon[112983]: 2023-06-22T17:27:56.902+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:57.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:56 smithi101 conmon[112983]: 2023-06-22T17:27:56.902+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:57.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:56 smithi101 conmon[112983]: 2023-06-22T17:27:56.902+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:57.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:56 smithi101 conmon[112983]: 2023-06-22T17:27:56.902+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.163707+0000 front 2023-06-22T17:27:04.163629+0000 (oldest deadline 2023-06-22T17:27:30.564301+0000) 2023-06-22T17:27:57.378 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:56 smithi101 conmon[112983]: 2023-06-22T17:27:56.902+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:14.065051+0000 front 2023-06-22T17:27:14.065033+0000 (oldest deadline 2023-06-22T17:27:39.365200+0000) 2023-06-22T17:27:57.378 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:56 smithi101 conmon[112983]: 2023-06-22T17:27:56.902+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:19.365904+0000 front 2023-06-22T17:27:19.365888+0000 (oldest deadline 2023-06-22T17:27:45.265802+0000) 2023-06-22T17:27:57.378 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:56 smithi101 conmon[112983]: 2023-06-22T17:27:56.902+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2023-06-22T17:27:29.867207+0000 front 2023-06-22T17:27:29.867162+0000 (oldest deadline 2023-06-22T17:27:55.766998+0000) 2023-06-22T17:27:57.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:56 smithi133 conmon[115049]: 2023-06-22T17:27:56.936+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:57.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:56 smithi133 conmon[115049]: 2023-06-22T17:27:56.936+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:57.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:56 smithi133 conmon[115049]: 2023-06-22T17:27:56.936+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:57.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:56 smithi133 conmon[115049]: 2023-06-22T17:27:56.936+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:13.936567+0000 front 2023-06-22T17:27:13.936510+0000 (oldest deadline 2023-06-22T17:27:36.236444+0000) 2023-06-22T17:27:57.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:56 smithi133 conmon[115049]: 2023-06-22T17:27:56.936+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:21.438006+0000 front 2023-06-22T17:27:21.438025+0000 (oldest deadline 2023-06-22T17:27:44.338268+0000) 2023-06-22T17:27:57.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:56 smithi133 conmon[115049]: 2023-06-22T17:27:56.936+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2023-06-22T17:27:29.039130+0000 front 2023-06-22T17:27:29.039276+0000 (oldest deadline 2023-06-22T17:27:51.939189+0000) 2023-06-22T17:27:58.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:57 smithi101 conmon[112983]: 2023-06-22T17:27:57.950+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:58.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:57 smithi101 conmon[112983]: 2023-06-22T17:27:57.950+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:58.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:57 smithi101 conmon[112983]: 2023-06-22T17:27:57.950+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:58.378 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:57 smithi101 conmon[112983]: 2023-06-22T17:27:57.950+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.163707+0000 front 2023-06-22T17:27:04.163629+0000 (oldest deadline 2023-06-22T17:27:30.564301+0000) 2023-06-22T17:27:58.378 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:57 smithi101 conmon[112983]: 2023-06-22T17:27:57.950+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:14.065051+0000 front 2023-06-22T17:27:14.065033+0000 (oldest deadline 2023-06-22T17:27:39.365200+0000) 2023-06-22T17:27:58.378 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:57 smithi101 conmon[112983]: 2023-06-22T17:27:57.950+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:19.365904+0000 front 2023-06-22T17:27:19.365888+0000 (oldest deadline 2023-06-22T17:27:45.265802+0000) 2023-06-22T17:27:58.378 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:57 smithi101 conmon[112983]: 2023-06-22T17:27:57.950+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2023-06-22T17:27:29.867207+0000 front 2023-06-22T17:27:29.867162+0000 (oldest deadline 2023-06-22T17:27:55.766998+0000) 2023-06-22T17:27:58.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:57 smithi133 conmon[115049]: 2023-06-22T17:27:57.986+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:58.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:57 smithi133 conmon[115049]: 2023-06-22T17:27:57.986+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:58.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:57 smithi133 conmon[115049]: 2023-06-22T17:27:57.986+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:58.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:57 smithi133 conmon[115049]: 2023-06-22T17:27:57.986+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:13.936567+0000 front 2023-06-22T17:27:13.936510+0000 (oldest deadline 2023-06-22T17:27:36.236444+0000) 2023-06-22T17:27:58.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:57 smithi133 conmon[115049]: 2023-06-22T17:27:57.986+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:21.438006+0000 front 2023-06-22T17:27:21.438025+0000 (oldest deadline 2023-06-22T17:27:44.338268+0000) 2023-06-22T17:27:58.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:57 smithi133 conmon[115049]: 2023-06-22T17:27:57.986+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2023-06-22T17:27:29.039130+0000 front 2023-06-22T17:27:29.039276+0000 (oldest deadline 2023-06-22T17:27:51.939189+0000) 2023-06-22T17:27:59.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:58 smithi101 conmon[112983]: 2023-06-22T17:27:58.953+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:27:59.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:58 smithi101 conmon[112983]: 2023-06-22T17:27:58.953+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:27:59.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:58 smithi101 conmon[112983]: 2023-06-22T17:27:58.953+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:27:59.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:58 smithi101 conmon[112983]: 2023-06-22T17:27:58.953+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.163707+0000 front 2023-06-22T17:27:04.163629+0000 (oldest deadline 2023-06-22T17:27:30.564301+0000) 2023-06-22T17:27:59.378 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:58 smithi101 conmon[112983]: 2023-06-22T17:27:58.953+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:14.065051+0000 front 2023-06-22T17:27:14.065033+0000 (oldest deadline 2023-06-22T17:27:39.365200+0000) 2023-06-22T17:27:59.378 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:58 smithi101 conmon[112983]: 2023-06-22T17:27:58.953+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:19.365904+0000 front 2023-06-22T17:27:19.365888+0000 (oldest deadline 2023-06-22T17:27:45.265802+0000) 2023-06-22T17:27:59.378 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:58 smithi101 conmon[112983]: 2023-06-22T17:27:58.953+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2023-06-22T17:27:29.867207+0000 front 2023-06-22T17:27:29.867162+0000 (oldest deadline 2023-06-22T17:27:55.766998+0000) 2023-06-22T17:27:59.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:59 smithi133 conmon[115049]: 2023-06-22T17:27:59.018+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:27:59.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:59 smithi133 conmon[115049]: 2023-06-22T17:27:59.018+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:27:59.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:59 smithi133 conmon[115049]: 2023-06-22T17:27:59.018+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:27:59.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:59 smithi133 conmon[115049]: 2023-06-22T17:27:59.018+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:13.936567+0000 front 2023-06-22T17:27:13.936510+0000 (oldest deadline 2023-06-22T17:27:36.236444+0000) 2023-06-22T17:27:59.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:59 smithi133 conmon[115049]: 2023-06-22T17:27:59.018+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:21.438006+0000 front 2023-06-22T17:27:21.438025+0000 (oldest deadline 2023-06-22T17:27:44.338268+0000) 2023-06-22T17:27:59.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:59 smithi133 conmon[115049]: 2023-06-22T17:27:59.018+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2023-06-22T17:27:29.039130+0000 front 2023-06-22T17:27:29.039276+0000 (oldest deadline 2023-06-22T17:27:51.939189+0000) 2023-06-22T17:28:00.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:59 smithi101 conmon[112983]: 2023-06-22T17:27:59.976+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:28:00.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:59 smithi101 conmon[112983]: 2023-06-22T17:27:59.977+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:28:00.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:59 smithi101 conmon[112983]: 2023-06-22T17:27:59.977+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:28:00.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:59 smithi101 conmon[112983]: 2023-06-22T17:27:59.977+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.163707+0000 front 2023-06-22T17:27:04.163629+0000 (oldest deadline 2023-06-22T17:27:30.564301+0000) 2023-06-22T17:28:00.378 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:59 smithi101 conmon[112983]: 2023-06-22T17:27:59.977+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:14.065051+0000 front 2023-06-22T17:27:14.065033+0000 (oldest deadline 2023-06-22T17:27:39.365200+0000) 2023-06-22T17:28:00.378 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:59 smithi101 conmon[112983]: 2023-06-22T17:27:59.977+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:19.365904+0000 front 2023-06-22T17:27:19.365888+0000 (oldest deadline 2023-06-22T17:27:45.265802+0000) 2023-06-22T17:28:00.378 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:27:59 smithi101 conmon[112983]: 2023-06-22T17:27:59.977+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2023-06-22T17:27:29.867207+0000 front 2023-06-22T17:27:29.867162+0000 (oldest deadline 2023-06-22T17:27:55.766998+0000) 2023-06-22T17:28:00.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:59 smithi133 conmon[115049]: 2023-06-22T17:27:59.969+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:28:00.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:59 smithi133 conmon[115049]: 2023-06-22T17:27:59.969+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:28:00.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:59 smithi133 conmon[115049]: 2023-06-22T17:27:59.969+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:28:00.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:59 smithi133 conmon[115049]: 2023-06-22T17:27:59.969+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:13.936567+0000 front 2023-06-22T17:27:13.936510+0000 (oldest deadline 2023-06-22T17:27:36.236444+0000) 2023-06-22T17:28:00.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:59 smithi133 conmon[115049]: 2023-06-22T17:27:59.969+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:21.438006+0000 front 2023-06-22T17:27:21.438025+0000 (oldest deadline 2023-06-22T17:27:44.338268+0000) 2023-06-22T17:28:00.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:27:59 smithi133 conmon[115049]: 2023-06-22T17:27:59.969+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2023-06-22T17:27:29.039130+0000 front 2023-06-22T17:27:29.039276+0000 (oldest deadline 2023-06-22T17:27:51.939189+0000) 2023-06-22T17:28:01.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:28:00 smithi101 conmon[112983]: 2023-06-22T17:28:00.963+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:44.559620+0000 front 2023-06-22T17:26:44.559479+0000 (oldest deadline 2023-06-22T17:27:09.259575+0000) 2023-06-22T17:28:01.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:28:00 smithi101 conmon[112983]: 2023-06-22T17:28:00.963+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:50.360600+0000 front 2023-06-22T17:26:50.360569+0000 (oldest deadline 2023-06-22T17:27:13.860740+0000) 2023-06-22T17:28:01.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:28:00 smithi101 conmon[112983]: 2023-06-22T17:28:00.963+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:58.462284+0000 front 2023-06-22T17:26:58.462406+0000 (oldest deadline 2023-06-22T17:27:21.362121+0000) 2023-06-22T17:28:01.378 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:28:00 smithi101 conmon[112983]: 2023-06-22T17:28:00.963+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2023-06-22T17:27:04.163707+0000 front 2023-06-22T17:27:04.163629+0000 (oldest deadline 2023-06-22T17:27:30.564301+0000) 2023-06-22T17:28:01.378 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:28:00 smithi101 conmon[112983]: 2023-06-22T17:28:00.963+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:14.065051+0000 front 2023-06-22T17:27:14.065033+0000 (oldest deadline 2023-06-22T17:27:39.365200+0000) 2023-06-22T17:28:01.378 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:28:00 smithi101 conmon[112983]: 2023-06-22T17:28:00.963+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:19.365904+0000 front 2023-06-22T17:27:19.365888+0000 (oldest deadline 2023-06-22T17:27:45.265802+0000) 2023-06-22T17:28:01.378 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:28:00 smithi101 conmon[112983]: 2023-06-22T17:28:00.963+0000 7fe920bdc700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2023-06-22T17:27:29.867207+0000 front 2023-06-22T17:27:29.867162+0000 (oldest deadline 2023-06-22T17:27:55.766998+0000) 2023-06-22T17:28:01.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:00 smithi133 conmon[115049]: 2023-06-22T17:28:00.993+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:28:01.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:00 smithi133 conmon[115049]: 2023-06-22T17:28:00.993+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:28:01.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:00 smithi133 conmon[115049]: 2023-06-22T17:28:00.993+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:28:01.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:00 smithi133 conmon[115049]: 2023-06-22T17:28:00.993+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:13.936567+0000 front 2023-06-22T17:27:13.936510+0000 (oldest deadline 2023-06-22T17:27:36.236444+0000) 2023-06-22T17:28:01.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:00 smithi133 conmon[115049]: 2023-06-22T17:28:00.993+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:21.438006+0000 front 2023-06-22T17:27:21.438025+0000 (oldest deadline 2023-06-22T17:27:44.338268+0000) 2023-06-22T17:28:01.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:00 smithi133 conmon[115049]: 2023-06-22T17:28:00.993+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2023-06-22T17:27:29.039130+0000 front 2023-06-22T17:27:29.039276+0000 (oldest deadline 2023-06-22T17:27:51.939189+0000) 2023-06-22T17:28:01.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:00 smithi133 conmon[115049]: 2023-06-22T17:28:00.993+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6820 osd.7 since back 2023-06-22T17:27:34.840481+0000 front 2023-06-22T17:27:34.840500+0000 (oldest deadline 2023-06-22T17:28:00.740464+0000) 2023-06-22T17:28:02.377 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jun 22 17:28:02 smithi101 bash[122827]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c-osd-10 2023-06-22T17:28:02.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:01 smithi133 conmon[115049]: 2023-06-22T17:28:01.971+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:28:02.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:01 smithi133 conmon[115049]: 2023-06-22T17:28:01.971+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:28:02.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:01 smithi133 conmon[115049]: 2023-06-22T17:28:01.971+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:28:02.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:01 smithi133 conmon[115049]: 2023-06-22T17:28:01.971+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:13.936567+0000 front 2023-06-22T17:27:13.936510+0000 (oldest deadline 2023-06-22T17:27:36.236444+0000) 2023-06-22T17:28:02.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:01 smithi133 conmon[115049]: 2023-06-22T17:28:01.971+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:21.438006+0000 front 2023-06-22T17:27:21.438025+0000 (oldest deadline 2023-06-22T17:27:44.338268+0000) 2023-06-22T17:28:02.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:01 smithi133 conmon[115049]: 2023-06-22T17:28:01.971+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2023-06-22T17:27:29.039130+0000 front 2023-06-22T17:27:29.039276+0000 (oldest deadline 2023-06-22T17:27:51.939189+0000) 2023-06-22T17:28:02.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:01 smithi133 conmon[115049]: 2023-06-22T17:28:01.971+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6820 osd.7 since back 2023-06-22T17:27:34.840481+0000 front 2023-06-22T17:27:34.840500+0000 (oldest deadline 2023-06-22T17:28:00.740464+0000) 2023-06-22T17:28:03.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:02 smithi133 conmon[115049]: 2023-06-22T17:28:02.921+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:28:03.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:02 smithi133 conmon[115049]: 2023-06-22T17:28:02.921+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:28:03.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:02 smithi133 conmon[115049]: 2023-06-22T17:28:02.921+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:28:03.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:02 smithi133 conmon[115049]: 2023-06-22T17:28:02.921+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:13.936567+0000 front 2023-06-22T17:27:13.936510+0000 (oldest deadline 2023-06-22T17:27:36.236444+0000) 2023-06-22T17:28:03.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:02 smithi133 conmon[115049]: 2023-06-22T17:28:02.921+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:21.438006+0000 front 2023-06-22T17:27:21.438025+0000 (oldest deadline 2023-06-22T17:27:44.338268+0000) 2023-06-22T17:28:03.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:02 smithi133 conmon[115049]: 2023-06-22T17:28:02.921+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2023-06-22T17:27:29.039130+0000 front 2023-06-22T17:27:29.039276+0000 (oldest deadline 2023-06-22T17:27:51.939189+0000) 2023-06-22T17:28:03.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:02 smithi133 conmon[115049]: 2023-06-22T17:28:02.921+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6820 osd.7 since back 2023-06-22T17:27:34.840481+0000 front 2023-06-22T17:27:34.840500+0000 (oldest deadline 2023-06-22T17:28:00.740464+0000) 2023-06-22T17:28:03.462 DEBUG:teuthology.orchestra.run.smithi101:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.10.service' 2023-06-22T17:28:03.501 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-22T17:28:03.501 INFO:tasks.cephadm.osd.10:Stopped osd.10 2023-06-22T17:28:03.502 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2023-06-22T17:28:03.502 DEBUG:teuthology.orchestra.run.smithi133:> sudo systemctl stop ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.11 2023-06-22T17:28:03.863 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:03 smithi133 systemd[1]: Stopping Ceph osd.11 for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:28:04.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:03 smithi133 conmon[115049]: 2023-06-22T17:28:03.861+0000 7f145196d700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-22T17:28:04.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:03 smithi133 conmon[115049]: 2023-06-22T17:28:03.861+0000 7f145196d700 -1 osd.11 81 *** Got signal Terminated *** 2023-06-22T17:28:04.171 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:03 smithi133 conmon[115049]: 2023-06-22T17:28:03.861+0000 7f145196d700 -1 osd.11 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-06-22T17:28:04.172 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:03 smithi133 conmon[115049]: 2023-06-22T17:28:03.963+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:28:04.172 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:03 smithi133 conmon[115049]: 2023-06-22T17:28:03.963+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:28:04.172 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:03 smithi133 conmon[115049]: 2023-06-22T17:28:03.963+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:28:04.172 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:03 smithi133 conmon[115049]: 2023-06-22T17:28:03.963+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:13.936567+0000 front 2023-06-22T17:27:13.936510+0000 (oldest deadline 2023-06-22T17:27:36.236444+0000) 2023-06-22T17:28:04.172 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:03 smithi133 conmon[115049]: 2023-06-22T17:28:03.963+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:21.438006+0000 front 2023-06-22T17:27:21.438025+0000 (oldest deadline 2023-06-22T17:27:44.338268+0000) 2023-06-22T17:28:04.172 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:03 smithi133 conmon[115049]: 2023-06-22T17:28:03.963+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2023-06-22T17:27:29.039130+0000 front 2023-06-22T17:27:29.039276+0000 (oldest deadline 2023-06-22T17:27:51.939189+0000) 2023-06-22T17:28:04.172 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:03 smithi133 conmon[115049]: 2023-06-22T17:28:03.963+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6820 osd.7 since back 2023-06-22T17:27:34.840481+0000 front 2023-06-22T17:27:34.840500+0000 (oldest deadline 2023-06-22T17:28:00.740464+0000) 2023-06-22T17:28:05.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:04 smithi133 conmon[115049]: 2023-06-22T17:28:04.987+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:28:05.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:04 smithi133 conmon[115049]: 2023-06-22T17:28:04.987+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:28:05.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:04 smithi133 conmon[115049]: 2023-06-22T17:28:04.987+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:28:05.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:04 smithi133 conmon[115049]: 2023-06-22T17:28:04.987+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:13.936567+0000 front 2023-06-22T17:27:13.936510+0000 (oldest deadline 2023-06-22T17:27:36.236444+0000) 2023-06-22T17:28:05.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:04 smithi133 conmon[115049]: 2023-06-22T17:28:04.987+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:21.438006+0000 front 2023-06-22T17:27:21.438025+0000 (oldest deadline 2023-06-22T17:27:44.338268+0000) 2023-06-22T17:28:05.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:04 smithi133 conmon[115049]: 2023-06-22T17:28:04.987+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2023-06-22T17:27:29.039130+0000 front 2023-06-22T17:27:29.039276+0000 (oldest deadline 2023-06-22T17:27:51.939189+0000) 2023-06-22T17:28:05.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:04 smithi133 conmon[115049]: 2023-06-22T17:28:04.987+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6820 osd.7 since back 2023-06-22T17:27:34.840481+0000 front 2023-06-22T17:27:34.840500+0000 (oldest deadline 2023-06-22T17:28:00.740464+0000) 2023-06-22T17:28:06.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:06 smithi133 conmon[115049]: 2023-06-22T17:28:06.032+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:28:06.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:06 smithi133 conmon[115049]: 2023-06-22T17:28:06.032+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:28:06.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:06 smithi133 conmon[115049]: 2023-06-22T17:28:06.032+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:28:06.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:06 smithi133 conmon[115049]: 2023-06-22T17:28:06.032+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:13.936567+0000 front 2023-06-22T17:27:13.936510+0000 (oldest deadline 2023-06-22T17:27:36.236444+0000) 2023-06-22T17:28:06.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:06 smithi133 conmon[115049]: 2023-06-22T17:28:06.032+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:21.438006+0000 front 2023-06-22T17:27:21.438025+0000 (oldest deadline 2023-06-22T17:27:44.338268+0000) 2023-06-22T17:28:06.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:06 smithi133 conmon[115049]: 2023-06-22T17:28:06.032+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2023-06-22T17:27:29.039130+0000 front 2023-06-22T17:27:29.039276+0000 (oldest deadline 2023-06-22T17:27:51.939189+0000) 2023-06-22T17:28:06.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:06 smithi133 conmon[115049]: 2023-06-22T17:28:06.032+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6820 osd.7 since back 2023-06-22T17:27:34.840481+0000 front 2023-06-22T17:27:34.840500+0000 (oldest deadline 2023-06-22T17:28:00.740464+0000) 2023-06-22T17:28:07.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:07 smithi133 conmon[115049]: 2023-06-22T17:28:07.034+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:28:07.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:07 smithi133 conmon[115049]: 2023-06-22T17:28:07.034+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:28:07.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:07 smithi133 conmon[115049]: 2023-06-22T17:28:07.034+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:28:07.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:07 smithi133 conmon[115049]: 2023-06-22T17:28:07.034+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:13.936567+0000 front 2023-06-22T17:27:13.936510+0000 (oldest deadline 2023-06-22T17:27:36.236444+0000) 2023-06-22T17:28:07.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:07 smithi133 conmon[115049]: 2023-06-22T17:28:07.034+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:21.438006+0000 front 2023-06-22T17:27:21.438025+0000 (oldest deadline 2023-06-22T17:27:44.338268+0000) 2023-06-22T17:28:07.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:07 smithi133 conmon[115049]: 2023-06-22T17:28:07.034+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2023-06-22T17:27:29.039130+0000 front 2023-06-22T17:27:29.039276+0000 (oldest deadline 2023-06-22T17:27:51.939189+0000) 2023-06-22T17:28:07.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:07 smithi133 conmon[115049]: 2023-06-22T17:28:07.034+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6820 osd.7 since back 2023-06-22T17:27:34.840481+0000 front 2023-06-22T17:27:34.840500+0000 (oldest deadline 2023-06-22T17:28:00.740464+0000) 2023-06-22T17:28:08.421 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:08 smithi133 conmon[115049]: 2023-06-22T17:28:08.005+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2023-06-22T17:26:43.730434+0000 front 2023-06-22T17:26:43.730607+0000 (oldest deadline 2023-06-22T17:27:06.630474+0000) 2023-06-22T17:28:08.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:08 smithi133 conmon[115049]: 2023-06-22T17:28:08.005+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-06-22T17:26:49.531827+0000 front 2023-06-22T17:26:49.531783+0000 (oldest deadline 2023-06-22T17:27:17.732296+0000) 2023-06-22T17:28:08.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:08 smithi133 conmon[115049]: 2023-06-22T17:28:08.005+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6804 osd.2 since back 2023-06-22T17:26:57.732718+0000 front 2023-06-22T17:26:57.732692+0000 (oldest deadline 2023-06-22T17:27:23.032888+0000) 2023-06-22T17:28:08.422 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:08 smithi133 conmon[115049]: 2023-06-22T17:28:08.005+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6812 osd.4 since back 2023-06-22T17:27:13.936567+0000 front 2023-06-22T17:27:13.936510+0000 (oldest deadline 2023-06-22T17:27:36.236444+0000) 2023-06-22T17:28:08.423 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:08 smithi133 conmon[115049]: 2023-06-22T17:28:08.005+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.133:6812 osd.5 since back 2023-06-22T17:27:21.438006+0000 front 2023-06-22T17:27:21.438025+0000 (oldest deadline 2023-06-22T17:27:44.338268+0000) 2023-06-22T17:28:08.423 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:08 smithi133 conmon[115049]: 2023-06-22T17:28:08.005+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.39:6822 osd.6 since back 2023-06-22T17:27:29.039130+0000 front 2023-06-22T17:27:29.039276+0000 (oldest deadline 2023-06-22T17:27:51.939189+0000) 2023-06-22T17:28:08.423 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:08 smithi133 conmon[115049]: 2023-06-22T17:28:08.005+0000 7f144ef6d700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.101:6820 osd.7 since back 2023-06-22T17:27:34.840481+0000 front 2023-06-22T17:27:34.840500+0000 (oldest deadline 2023-06-22T17:28:00.740464+0000) 2023-06-22T17:28:09.921 INFO:journalctl@ceph.osd.11.smithi133.stdout:Jun 22 17:28:09 smithi133 bash[125198]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c-osd-11 2023-06-22T17:28:11.066 DEBUG:teuthology.orchestra.run.smithi133:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@osd.11.service' 2023-06-22T17:28:11.103 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-22T17:28:11.104 INFO:tasks.cephadm.osd.11:Stopped osd.11 2023-06-22T17:28:11.104 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2023-06-22T17:28:11.105 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.a 2023-06-22T17:28:11.434 INFO:journalctl@ceph.mds.a.smithi039.stdout:Jun 22 17:28:11 smithi039 systemd[1]: Stopping Ceph mds.a for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:28:11.934 INFO:journalctl@ceph.mds.a.smithi039.stdout:Jun 22 17:28:11 smithi039 conmon[130031]: 2023-06-22T17:28:11.455+0000 7fe9275a4700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-22T17:28:11.934 INFO:journalctl@ceph.mds.a.smithi039.stdout:Jun 22 17:28:11 smithi039 conmon[130031]: 2023-06-22T17:28:11.455+0000 7fe9275a4700 -1 mds.a *** got signal Terminated *** 2023-06-22T17:28:13.007 INFO:journalctl@ceph.mds.a.smithi039.stdout:Jun 22 17:28:12 smithi039 bash[135721]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c-mds-a 2023-06-22T17:28:13.304 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.a.service' 2023-06-22T17:28:13.340 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-22T17:28:13.341 INFO:tasks.cephadm.mds.a:Stopped mds.a 2023-06-22T17:28:13.342 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2023-06-22T17:28:13.342 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.d 2023-06-22T17:28:13.684 INFO:journalctl@ceph.mds.d.smithi039.stdout:Jun 22 17:28:13 smithi039 systemd[1]: Stopping Ceph mds.d for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:28:14.184 INFO:journalctl@ceph.mds.d.smithi039.stdout:Jun 22 17:28:13 smithi039 conmon[128614]: 2023-06-22T17:28:13.713+0000 7fa06b817700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-22T17:28:14.184 INFO:journalctl@ceph.mds.d.smithi039.stdout:Jun 22 17:28:13 smithi039 conmon[128614]: 2023-06-22T17:28:13.713+0000 7fa06b817700 -1 mds.d *** got signal Terminated *** 2023-06-22T17:28:17.184 INFO:journalctl@ceph.mds.d.smithi039.stdout:Jun 22 17:28:16 smithi039 bash[136013]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c-mds-d 2023-06-22T17:28:17.224 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.d.service' 2023-06-22T17:28:17.260 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-22T17:28:17.261 INFO:tasks.cephadm.mds.d:Stopped mds.d 2023-06-22T17:28:17.261 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2023-06-22T17:28:17.262 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.g 2023-06-22T17:28:17.649 INFO:journalctl@ceph.mds.g.smithi039.stdout:Jun 22 17:28:17 smithi039 systemd[1]: Stopping Ceph mds.g for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:28:17.934 INFO:journalctl@ceph.mds.g.smithi039.stdout:Jun 22 17:28:17 smithi039 conmon[129431]: 2023-06-22T17:28:17.646+0000 7f5acf61e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-22T17:28:17.934 INFO:journalctl@ceph.mds.g.smithi039.stdout:Jun 22 17:28:17 smithi039 conmon[129431]: 2023-06-22T17:28:17.646+0000 7f5acf61e700 -1 mds.g *** got signal Terminated *** 2023-06-22T17:28:22.139 INFO:journalctl@ceph.mds.g.smithi039.stdout:Jun 22 17:28:21 smithi039 bash[136290]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c-mds-g 2023-06-22T17:28:22.146 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.g.service' 2023-06-22T17:28:22.180 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-22T17:28:22.181 INFO:tasks.cephadm.mds.g:Stopped mds.g 2023-06-22T17:28:22.181 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2023-06-22T17:28:22.182 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.j 2023-06-22T17:28:22.434 INFO:journalctl@ceph.mds.j.smithi039.stdout:Jun 22 17:28:22 smithi039 systemd[1]: Stopping Ceph mds.j for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:28:22.934 INFO:journalctl@ceph.mds.j.smithi039.stdout:Jun 22 17:28:22 smithi039 conmon[128031]: 2023-06-22T17:28:22.669+0000 7f4cf3e36700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-22T17:28:22.934 INFO:journalctl@ceph.mds.j.smithi039.stdout:Jun 22 17:28:22 smithi039 conmon[128031]: 2023-06-22T17:28:22.669+0000 7f4cf3e36700 -1 mds.j *** got signal Terminated *** 2023-06-22T17:28:27.084 INFO:journalctl@ceph.mds.j.smithi039.stdout:Jun 22 17:28:26 smithi039 bash[136565]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c-mds-j 2023-06-22T17:28:27.089 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.j.service' 2023-06-22T17:28:27.122 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-22T17:28:27.123 INFO:tasks.cephadm.mds.j:Stopped mds.j 2023-06-22T17:28:27.123 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2023-06-22T17:28:27.123 DEBUG:teuthology.orchestra.run.smithi101:> sudo systemctl stop ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.b 2023-06-22T17:28:27.478 INFO:journalctl@ceph.mds.b.smithi101.stdout:Jun 22 17:28:27 smithi101 systemd[1]: Stopping Ceph mds.b for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:28:27.877 INFO:journalctl@ceph.mds.b.smithi101.stdout:Jun 22 17:28:27 smithi101 conmon[118170]: 2023-06-22T17:28:27.475+0000 7f1a92b31700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-22T17:28:27.877 INFO:journalctl@ceph.mds.b.smithi101.stdout:Jun 22 17:28:27 smithi101 conmon[118170]: 2023-06-22T17:28:27.475+0000 7f1a92b31700 -1 mds.b *** got signal Terminated *** 2023-06-22T17:28:31.300 INFO:journalctl@ceph.mds.b.smithi101.stdout:Jun 22 17:28:30 smithi101 bash[123369]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c-mds-b 2023-06-22T17:28:31.306 DEBUG:teuthology.orchestra.run.smithi101:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.b.service' 2023-06-22T17:28:31.341 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-22T17:28:31.341 INFO:tasks.cephadm.mds.b:Stopped mds.b 2023-06-22T17:28:31.342 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2023-06-22T17:28:31.342 DEBUG:teuthology.orchestra.run.smithi101:> sudo systemctl stop ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.e 2023-06-22T17:28:31.627 INFO:journalctl@ceph.mds.e.smithi101.stdout:Jun 22 17:28:31 smithi101 systemd[1]: Stopping Ceph mds.e for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:28:32.127 INFO:journalctl@ceph.mds.e.smithi101.stdout:Jun 22 17:28:31 smithi101 conmon[117043]: 2023-06-22T17:28:31.715+0000 7f4abfc20700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-22T17:28:32.127 INFO:journalctl@ceph.mds.e.smithi101.stdout:Jun 22 17:28:31 smithi101 conmon[117043]: 2023-06-22T17:28:31.715+0000 7f4abfc20700 -1 mds.e *** got signal Terminated *** 2023-06-22T17:28:35.307 INFO:journalctl@ceph.mds.e.smithi101.stdout:Jun 22 17:28:35 smithi101 bash[123654]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c-mds-e 2023-06-22T17:28:35.313 DEBUG:teuthology.orchestra.run.smithi101:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.e.service' 2023-06-22T17:28:35.345 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-22T17:28:35.346 INFO:tasks.cephadm.mds.e:Stopped mds.e 2023-06-22T17:28:35.346 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2023-06-22T17:28:35.347 DEBUG:teuthology.orchestra.run.smithi101:> sudo systemctl stop ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.h 2023-06-22T17:28:35.627 INFO:journalctl@ceph.mds.h.smithi101.stdout:Jun 22 17:28:35 smithi101 systemd[1]: Stopping Ceph mds.h for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:28:36.127 INFO:journalctl@ceph.mds.h.smithi101.stdout:Jun 22 17:28:35 smithi101 conmon[117605]: 2023-06-22T17:28:35.716+0000 7f1c317a1700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-22T17:28:36.127 INFO:journalctl@ceph.mds.h.smithi101.stdout:Jun 22 17:28:35 smithi101 conmon[117605]: 2023-06-22T17:28:35.716+0000 7f1c317a1700 -1 mds.h *** got signal Terminated *** 2023-06-22T17:28:38.306 INFO:journalctl@ceph.mds.h.smithi101.stdout:Jun 22 17:28:38 smithi101 bash[123934]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c-mds-h 2023-06-22T17:28:38.312 DEBUG:teuthology.orchestra.run.smithi101:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.h.service' 2023-06-22T17:28:38.342 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-22T17:28:38.343 INFO:tasks.cephadm.mds.h:Stopped mds.h 2023-06-22T17:28:38.344 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2023-06-22T17:28:38.344 DEBUG:teuthology.orchestra.run.smithi101:> sudo systemctl stop ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.k 2023-06-22T17:28:38.627 INFO:journalctl@ceph.mds.k.smithi101.stdout:Jun 22 17:28:38 smithi101 systemd[1]: Stopping Ceph mds.k for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:28:39.127 INFO:journalctl@ceph.mds.k.smithi101.stdout:Jun 22 17:28:38 smithi101 conmon[116484]: 2023-06-22T17:28:38.723+0000 7f81c878c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-22T17:28:39.127 INFO:journalctl@ceph.mds.k.smithi101.stdout:Jun 22 17:28:38 smithi101 conmon[116484]: 2023-06-22T17:28:38.723+0000 7f81c878c700 -1 mds.k *** got signal Terminated *** 2023-06-22T17:28:40.821 INFO:journalctl@ceph.mds.k.smithi101.stdout:Jun 22 17:28:40 smithi101 bash[124211]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c-mds-k 2023-06-22T17:28:40.828 DEBUG:teuthology.orchestra.run.smithi101:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.k.service' 2023-06-22T17:28:40.858 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-22T17:28:40.858 INFO:tasks.cephadm.mds.k:Stopped mds.k 2023-06-22T17:28:40.859 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2023-06-22T17:28:40.859 DEBUG:teuthology.orchestra.run.smithi133:> sudo systemctl stop ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.c 2023-06-22T17:28:41.171 INFO:journalctl@ceph.mds.c.smithi133.stdout:Jun 22 17:28:40 smithi133 systemd[1]: Stopping Ceph mds.c for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:28:41.671 INFO:journalctl@ceph.mds.c.smithi133.stdout:Jun 22 17:28:41 smithi133 conmon[120011]: 2023-06-22T17:28:41.211+0000 7fa4469f8700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-22T17:28:41.671 INFO:journalctl@ceph.mds.c.smithi133.stdout:Jun 22 17:28:41 smithi133 conmon[120011]: 2023-06-22T17:28:41.211+0000 7fa4469f8700 -1 mds.c *** got signal Terminated *** 2023-06-22T17:28:43.922 INFO:journalctl@ceph.mds.c.smithi133.stdout:Jun 22 17:28:43 smithi133 bash[125741]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c-mds-c 2023-06-22T17:28:43.965 DEBUG:teuthology.orchestra.run.smithi133:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.c.service' 2023-06-22T17:28:44.000 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-22T17:28:44.001 INFO:tasks.cephadm.mds.c:Stopped mds.c 2023-06-22T17:28:44.001 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2023-06-22T17:28:44.002 DEBUG:teuthology.orchestra.run.smithi133:> sudo systemctl stop ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.f 2023-06-22T17:28:44.390 INFO:journalctl@ceph.mds.f.smithi133.stdout:Jun 22 17:28:44 smithi133 systemd[1]: Stopping Ceph mds.f for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:28:44.671 INFO:journalctl@ceph.mds.f.smithi133.stdout:Jun 22 17:28:44 smithi133 conmon[119449]: 2023-06-22T17:28:44.386+0000 7f0908fff700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-22T17:28:44.672 INFO:journalctl@ceph.mds.f.smithi133.stdout:Jun 22 17:28:44 smithi133 conmon[119449]: 2023-06-22T17:28:44.386+0000 7f0908fff700 -1 mds.f *** got signal Terminated *** 2023-06-22T17:28:48.872 INFO:journalctl@ceph.mds.f.smithi133.stdout:Jun 22 17:28:48 smithi133 bash[126030]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c-mds-f 2023-06-22T17:28:48.879 DEBUG:teuthology.orchestra.run.smithi133:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.f.service' 2023-06-22T17:28:48.911 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-22T17:28:48.912 INFO:tasks.cephadm.mds.f:Stopped mds.f 2023-06-22T17:28:48.912 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2023-06-22T17:28:48.913 DEBUG:teuthology.orchestra.run.smithi133:> sudo systemctl stop ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.i 2023-06-22T17:28:49.171 INFO:journalctl@ceph.mds.i.smithi133.stdout:Jun 22 17:28:48 smithi133 systemd[1]: Stopping Ceph mds.i for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:28:49.671 INFO:journalctl@ceph.mds.i.smithi133.stdout:Jun 22 17:28:49 smithi133 conmon[120573]: 2023-06-22T17:28:49.301+0000 7fd982ced700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-22T17:28:49.672 INFO:journalctl@ceph.mds.i.smithi133.stdout:Jun 22 17:28:49 smithi133 conmon[120573]: 2023-06-22T17:28:49.301+0000 7fd982ced700 -1 mds.i *** got signal Terminated *** 2023-06-22T17:28:58.635 INFO:journalctl@ceph.mds.i.smithi133.stdout:Jun 22 17:28:58 smithi133 bash[126308]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c-mds-i 2023-06-22T17:28:58.797 DEBUG:teuthology.orchestra.run.smithi133:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.i.service' 2023-06-22T17:28:58.829 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-22T17:28:58.830 INFO:tasks.cephadm.mds.i:Stopped mds.i 2023-06-22T17:28:58.830 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2023-06-22T17:28:58.831 DEBUG:teuthology.orchestra.run.smithi133:> sudo systemctl stop ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.l 2023-06-22T17:28:59.171 INFO:journalctl@ceph.mds.l.smithi133.stdout:Jun 22 17:28:58 smithi133 systemd[1]: Stopping Ceph mds.l for d88002e2-1120-11ee-9b2b-001a4aab830c... 2023-06-22T17:28:59.671 INFO:journalctl@ceph.mds.l.smithi133.stdout:Jun 22 17:28:59 smithi133 conmon[118886]: 2023-06-22T17:28:59.200+0000 7f5703bc3700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-22T17:28:59.671 INFO:journalctl@ceph.mds.l.smithi133.stdout:Jun 22 17:28:59 smithi133 conmon[118886]: 2023-06-22T17:28:59.200+0000 7f5703bc3700 -1 mds.l *** got signal Terminated *** 2023-06-22T17:29:02.622 INFO:journalctl@ceph.mds.l.smithi133.stdout:Jun 22 17:29:02 smithi133 bash[126587]: ceph-d88002e2-1120-11ee-9b2b-001a4aab830c-mds-l 2023-06-22T17:29:02.707 DEBUG:teuthology.orchestra.run.smithi133:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d88002e2-1120-11ee-9b2b-001a4aab830c@mds.l.service' 2023-06-22T17:29:02.738 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-22T17:29:02.739 INFO:tasks.cephadm.mds.l:Stopped mds.l 2023-06-22T17:29:02.740 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d88002e2-1120-11ee-9b2b-001a4aab830c --force --keep-logs 2023-06-22T17:29:04.816 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d88002e2-1120-11ee-9b2b-001a4aab830c --force --keep-logs 2023-06-22T17:29:06.796 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d88002e2-1120-11ee-9b2b-001a4aab830c --force --keep-logs 2023-06-22T17:29:08.761 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-06-22T17:29:08.793 DEBUG:teuthology.orchestra.run.smithi101:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-06-22T17:29:08.825 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-06-22T17:29:08.852 INFO:tasks.cephadm:Archiving crash dumps... 2023-06-22T17:29:08.855 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/crash to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311823/remote/smithi039/crash 2023-06-22T17:29:08.868 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/crash -- . 2023-06-22T17:29:08.905 INFO:teuthology.orchestra.run.smithi039.stderr:tar: /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/crash: Cannot open: No such file or directory 2023-06-22T17:29:08.906 INFO:teuthology.orchestra.run.smithi039.stderr:tar: Error is not recoverable: exiting now 2023-06-22T17:29:08.908 DEBUG:teuthology.misc:Transferring archived files from smithi101:/var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/crash to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311823/remote/smithi101/crash 2023-06-22T17:29:08.910 DEBUG:teuthology.orchestra.run.smithi101:> sudo tar c -f - -C /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/crash -- . 2023-06-22T17:29:08.953 INFO:teuthology.orchestra.run.smithi101.stderr:tar: /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/crash: Cannot open: No such file or directory 2023-06-22T17:29:08.953 INFO:teuthology.orchestra.run.smithi101.stderr:tar: Error is not recoverable: exiting now 2023-06-22T17:29:08.955 DEBUG:teuthology.misc:Transferring archived files from smithi133:/var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/crash to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311823/remote/smithi133/crash 2023-06-22T17:29:08.957 DEBUG:teuthology.orchestra.run.smithi133:> sudo tar c -f - -C /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/crash -- . 2023-06-22T17:29:09.002 INFO:teuthology.orchestra.run.smithi133.stderr:tar: /var/lib/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/crash: Cannot open: No such file or directory 2023-06-22T17:29:09.002 INFO:teuthology.orchestra.run.smithi133.stderr:tar: Error is not recoverable: exiting now 2023-06-22T17:29:09.003 INFO:tasks.cephadm:Checking cluster log for badness... 2023-06-22T17:29:09.004 DEBUG:teuthology.orchestra.run.smithi039:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-06-22T17:29:09.035 INFO:teuthology.orchestra.run.smithi039.stderr:grep: /var/log/ceph/d88002e2-1120-11ee-9b2b-001a4aab830c/ceph.log: No such file or directory 2023-06-22T17:29:09.037 INFO:tasks.cephadm:Compressing logs... 2023-06-22T17:29:09.038 DEBUG:teuthology.orchestra.run.smithi039:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-22T17:29:09.080 DEBUG:teuthology.orchestra.run.smithi101:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-22T17:29:09.082 DEBUG:teuthology.orchestra.run.smithi133:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-22T17:29:09.104 INFO:teuthology.orchestra.run.smithi039.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-06-22T17:29:09.107 INFO:teuthology.orchestra.run.smithi133.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-06-22T17:29:09.109 INFO:teuthology.orchestra.run.smithi101.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-06-22T17:29:09.931 INFO:tasks.cephadm:Archiving logs... 2023-06-22T17:29:09.932 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311823/remote/smithi039/log 2023-06-22T17:29:09.934 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /var/log/ceph -- . 2023-06-22T17:29:10.097 DEBUG:teuthology.misc:Transferring archived files from smithi101:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311823/remote/smithi101/log 2023-06-22T17:29:10.098 DEBUG:teuthology.orchestra.run.smithi101:> sudo tar c -f - -C /var/log/ceph -- . 2023-06-22T17:29:10.187 DEBUG:teuthology.misc:Transferring archived files from smithi133:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311823/remote/smithi133/log 2023-06-22T17:29:10.188 DEBUG:teuthology.orchestra.run.smithi133:> sudo tar c -f - -C /var/log/ceph -- . 2023-06-22T17:29:10.271 INFO:tasks.cephadm:Removing cluster... 2023-06-22T17:29:10.272 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d88002e2-1120-11ee-9b2b-001a4aab830c --force 2023-06-22T17:29:11.075 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d88002e2-1120-11ee-9b2b-001a4aab830c --force 2023-06-22T17:29:11.853 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d88002e2-1120-11ee-9b2b-001a4aab830c --force 2023-06-22T17:29:12.646 INFO:tasks.cephadm:Removing cephadm ... 2023-06-22T17:29:12.647 DEBUG:teuthology.orchestra.run.smithi039:> rm -rf /home/ubuntu/cephtest/cephadm 2023-06-22T17:29:12.663 DEBUG:teuthology.orchestra.run.smithi101:> rm -rf /home/ubuntu/cephtest/cephadm 2023-06-22T17:29:12.679 DEBUG:teuthology.orchestra.run.smithi133:> rm -rf /home/ubuntu/cephtest/cephadm 2023-06-22T17:29:12.694 INFO:tasks.cephadm:Teardown complete 2023-06-22T17:29:12.695 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephadm.py", line 1561, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephadm.py", line 1474, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephadm.py", line 896, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 514, in __init__ self.create() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 717, in create self.set_allow_standby_replay(standby_replay) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 589, in set_allow_standby_replay self.set_var("allow_standby_replay", yes) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 574, in set_var self.mon_manager.raw_cluster_cmd("fs", "set", self.name, var, *a) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1624, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1615, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi039 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true' 2023-06-22T17:29:12.773 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=abde005e46504e8cafffb0772f206a1e Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephadm.py", line 1561, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephadm.py", line 1474, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephadm.py", line 896, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 514, in __init__ self.create() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 717, in create self.set_allow_standby_replay(standby_replay) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 589, in set_allow_standby_replay self.set_var("allow_standby_replay", yes) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 574, in set_var self.mon_manager.raw_cluster_cmd("fs", "set", self.name, var, *a) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1624, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1615, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi039 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true' 2023-06-22T17:29:12.779 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-06-22T17:29:12.789 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-06-22T17:29:12.830 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-06-22T17:29:12.831 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-06-22T17:29:12.859 DEBUG:teuthology.orchestra.run.smithi101:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-06-22T17:29:12.863 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-06-22T17:29:12.963 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-06-22T17:29:12.972 INFO:teuthology.task.clock:Checking final clock skew... 2023-06-22T17:29:12.973 DEBUG:teuthology.orchestra.run.smithi039:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-22T17:29:12.975 DEBUG:teuthology.orchestra.run.smithi101:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-22T17:29:12.978 DEBUG:teuthology.orchestra.run.smithi133:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-22T17:29:12.989 INFO:teuthology.orchestra.run.smithi039.stderr:bash: ntpq: command not found 2023-06-22T17:29:12.990 INFO:teuthology.orchestra.run.smithi101.stderr:bash: ntpq: command not found 2023-06-22T17:29:12.991 INFO:teuthology.orchestra.run.smithi133.stderr:bash: ntpq: command not found 2023-06-22T17:29:12.993 INFO:teuthology.orchestra.run.smithi101.stdout:210 Number of sources = 4 2023-06-22T17:29:12.993 INFO:teuthology.orchestra.run.smithi101.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-22T17:29:12.993 INFO:teuthology.orchestra.run.smithi101.stdout:=============================================================================== 2023-06-22T17:29:12.993 INFO:teuthology.orchestra.run.smithi101.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 59 +1499us[+1499us] +/- 113ms 2023-06-22T17:29:12.994 INFO:teuthology.orchestra.run.smithi101.stdout:^* hv02.front.sepia.ceph.com 3 6 377 62 -2873us[-2831us] +/- 60ms 2023-06-22T17:29:12.994 INFO:teuthology.orchestra.run.smithi101.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 60 +2403us[+2403us] +/- 69ms 2023-06-22T17:29:12.994 INFO:teuthology.orchestra.run.smithi101.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T17:29:12.996 INFO:teuthology.orchestra.run.smithi133.stdout:210 Number of sources = 4 2023-06-22T17:29:12.996 INFO:teuthology.orchestra.run.smithi133.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-22T17:29:12.996 INFO:teuthology.orchestra.run.smithi133.stdout:=============================================================================== 2023-06-22T17:29:12.996 INFO:teuthology.orchestra.run.smithi133.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 62 +1508us[+1530us] +/- 113ms 2023-06-22T17:29:12.997 INFO:teuthology.orchestra.run.smithi133.stdout:^* hv02.front.sepia.ceph.com 3 6 377 61 -2870us[-2848us] +/- 60ms 2023-06-22T17:29:12.997 INFO:teuthology.orchestra.run.smithi133.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 60 +2414us[+2414us] +/- 69ms 2023-06-22T17:29:12.997 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T17:29:12.997 INFO:teuthology.orchestra.run.smithi039.stdout:210 Number of sources = 4 2023-06-22T17:29:12.998 INFO:teuthology.orchestra.run.smithi039.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-22T17:29:12.998 INFO:teuthology.orchestra.run.smithi039.stdout:=============================================================================== 2023-06-22T17:29:12.998 INFO:teuthology.orchestra.run.smithi039.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 61 +1530us[+1551us] +/- 113ms 2023-06-22T17:29:12.998 INFO:teuthology.orchestra.run.smithi039.stdout:^* hv02.front.sepia.ceph.com 3 6 377 60 -2851us[-2829us] +/- 60ms 2023-06-22T17:29:12.998 INFO:teuthology.orchestra.run.smithi039.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 61 +2432us[+2453us] +/- 69ms 2023-06-22T17:29:12.999 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T17:29:12.999 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-06-22T17:29:13.008 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-06-22T17:29:13.009 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-06-22T17:29:13.017 DEBUG:teuthology.orchestra.run.smithi039:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-06-22T17:29:13.135 DEBUG:teuthology.orchestra.run.smithi101:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-06-22T17:29:13.225 DEBUG:teuthology.orchestra.run.smithi133:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-06-22T17:29:13.313 DEBUG:teuthology.orchestra.run.smithi039:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-22T17:29:13.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-22T17:29:13.346 DEBUG:teuthology.orchestra.run.smithi101:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-22T17:29:13.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-22T17:29:13.376 DEBUG:teuthology.orchestra.run.smithi133:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-22T17:29:13.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-22T17:29:13.406 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-06-22T17:29:13.421 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-06-22T17:29:13.460 INFO:teuthology.task.internal:Duration was 1255.772494 seconds 2023-06-22T17:29:13.461 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-06-22T17:29:13.499 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-06-22T17:29:13.500 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-06-22T17:29:13.502 DEBUG:teuthology.orchestra.run.smithi101:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-06-22T17:29:13.504 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-06-22T17:29:13.546 INFO:teuthology.orchestra.run.smithi101.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-22T17:29:13.547 INFO:teuthology.orchestra.run.smithi039.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-22T17:29:13.548 INFO:teuthology.orchestra.run.smithi133.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-22T17:29:13.933 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-06-22T17:29:13.934 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi039.front.sepia.ceph.com 2023-06-22T17:29:13.934 DEBUG:teuthology.orchestra.run.smithi039:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-06-22T17:29:13.955 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi101.front.sepia.ceph.com 2023-06-22T17:29:13.955 DEBUG:teuthology.orchestra.run.smithi101:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-06-22T17:29:13.975 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi133.front.sepia.ceph.com 2023-06-22T17:29:13.976 DEBUG:teuthology.orchestra.run.smithi133:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-06-22T17:29:13.996 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-06-22T17:29:13.996 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-22T17:29:13.999 DEBUG:teuthology.orchestra.run.smithi101:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-22T17:29:14.018 DEBUG:teuthology.orchestra.run.smithi133:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-22T17:29:14.079 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-06-22T17:29:14.079 DEBUG:teuthology.orchestra.run.smithi039:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-06-22T17:29:14.082 DEBUG:teuthology.orchestra.run.smithi101:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-06-22T17:29:14.100 DEBUG:teuthology.orchestra.run.smithi133:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-06-22T17:29:14.210 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-06-22T17:29:14.220 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-06-22T17:29:14.221 DEBUG:teuthology.orchestra.run.smithi039:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-06-22T17:29:14.246 DEBUG:teuthology.orchestra.run.smithi101:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-06-22T17:29:14.271 DEBUG:teuthology.orchestra.run.smithi133:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-06-22T17:29:14.297 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-06-22T17:29:14.315 DEBUG:teuthology.orchestra.run.smithi039:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-06-22T17:29:14.318 DEBUG:teuthology.orchestra.run.smithi101:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-06-22T17:29:14.320 DEBUG:teuthology.orchestra.run.smithi133:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-06-22T17:29:14.340 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = core 2023-06-22T17:29:14.342 INFO:teuthology.orchestra.run.smithi101.stdout:kernel.core_pattern = core 2023-06-22T17:29:14.360 INFO:teuthology.orchestra.run.smithi133.stdout:kernel.core_pattern = core 2023-06-22T17:29:14.375 DEBUG:teuthology.orchestra.run.smithi039:> test -e /home/ubuntu/cephtest/archive/coredump 2023-06-22T17:29:14.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-22T17:29:14.410 DEBUG:teuthology.orchestra.run.smithi101:> test -e /home/ubuntu/cephtest/archive/coredump 2023-06-22T17:29:14.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-22T17:29:14.425 DEBUG:teuthology.orchestra.run.smithi133:> test -e /home/ubuntu/cephtest/archive/coredump 2023-06-22T17:29:14.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-22T17:29:14.440 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-06-22T17:29:14.453 INFO:teuthology.task.internal:Transferring archived files... 2023-06-22T17:29:14.454 DEBUG:teuthology.misc:Transferring archived files from smithi039:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311823/remote/smithi039 2023-06-22T17:29:14.454 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-06-22T17:29:14.490 DEBUG:teuthology.misc:Transferring archived files from smithi101:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311823/remote/smithi101 2023-06-22T17:29:14.491 DEBUG:teuthology.orchestra.run.smithi101:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-06-22T17:29:14.529 DEBUG:teuthology.misc:Transferring archived files from smithi133:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311823/remote/smithi133 2023-06-22T17:29:14.530 DEBUG:teuthology.orchestra.run.smithi133:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-06-22T17:29:14.565 INFO:teuthology.task.internal:Removing archive directory... 2023-06-22T17:29:14.565 DEBUG:teuthology.orchestra.run.smithi039:> rm -rf -- /home/ubuntu/cephtest/archive 2023-06-22T17:29:14.568 DEBUG:teuthology.orchestra.run.smithi101:> rm -rf -- /home/ubuntu/cephtest/archive 2023-06-22T17:29:14.571 DEBUG:teuthology.orchestra.run.smithi133:> rm -rf -- /home/ubuntu/cephtest/archive 2023-06-22T17:29:14.613 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-06-22T17:29:14.634 INFO:teuthology.task.internal:Not uploading archives. 2023-06-22T17:29:14.635 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-06-22T17:29:14.672 INFO:teuthology.task.internal:Tidying up after the test... 2023-06-22T17:29:14.673 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-06-22T17:29:14.676 DEBUG:teuthology.orchestra.run.smithi101:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-06-22T17:29:14.678 DEBUG:teuthology.orchestra.run.smithi133:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-06-22T17:29:14.688 INFO:teuthology.orchestra.run.smithi039.stdout: 262153 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 22 17:29 /home/ubuntu/cephtest 2023-06-22T17:29:14.690 INFO:teuthology.orchestra.run.smithi101.stdout: 262153 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 22 17:29 /home/ubuntu/cephtest 2023-06-22T17:29:14.692 INFO:teuthology.orchestra.run.smithi133.stdout: 262153 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 22 17:29 /home/ubuntu/cephtest 2023-06-22T17:29:14.693 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-06-22T17:29:14.733 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-06-22T17:29:14.764 INFO:teuthology.nuke:Checking targets against current locks 2023-06-22T17:29:14.797 DEBUG:teuthology.nuke:shortname: smithi039 2023-06-22T17:29:14.798 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-22T17:29:14.823 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi039.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311823', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-22 16:58:52.438243', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZePTAHvZbx6OtHBjisBZNYiX7R2eUrvpKqn+Br+kxucLXXbrPLIiyoul/s4+IiB6NnHtVu7M+igECv+GnMtvYW5y5TzivzjheGR2LeVgFfDJqqPvsmEnwNJoWfWLOukfpwG6aZPUmo0jBvsEci3DqgG1DmkwCTvh+oRxHeWsKvMaTASKYgM1a6ww25p7y3mjHPetaI1ca7YNLPpkiZc4XHPNfkyUO0suFsQbfK9lVqIbUq0yKnMfB0jAr1l5tQkwUX6+KMaFNS+tW2jwt+vh93hxWjvvjeUB9F2A+fhAZbRoeDd4pujduoXtC6OfWFvLQJfkzb1wQfyFHam1BoOSRaTw5E2MsOSv0PCYJnZ55PRt94caNrDQwVKQ+3r1eonhOcAFJmh58wpL8EolFQ9WGHSxMY8Xeb3Y0fLjykOa+W3QSdqZQC+1E0GGEj4+NKmRvMrKLzGyyoyehGFiEHIjkiYgEe0gsPk+C4XO3aPxeC/R0lz9fMUgo/CtpRixlu8M='} 2023-06-22T17:29:14.827 DEBUG:teuthology.nuke:shortname: smithi101 2023-06-22T17:29:14.828 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-22T17:29:14.855 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi101.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311823', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-22 16:58:52.436353', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCUPiLlj4GcVhnjueUMi4vjBDm/pnWv0LBKz/KVj6tO1gukUoK2H12nzaWpbkwGVTSHKsjiAjeCt6kOtOOiyLZ+ZsdDQ6C7puyz55nSLjSQ7Am/ZVR6W988kRDKTzNhT0SbxcfEQcp7Q3eiA4v4keIBKdVJKnJDP3z/jEVH0Q8oAhSvOebfa6ArD2yc78QRbcnjHp+97O6qYUhzBM236l+gTP+x9E4GCk9t0pP7y/5bFxiSr2eYkNyFSfDJQGkeVsobpoTTMKze8QANo0vwxGLqYNjlS/EDNSLYW/mEzX2nSO0tXYRPWwW54mH7H7m3GACQ2xC6+AfD8vtKmIQNvXagzMqSPIZa4moVRLEkmoWJWibHpOw+DPEEVR5LwzICUaxM2PCRq4YVmVKFP8/FyaJMr7RvWrDIOZvF93oZBEE5cJmAC9s5Bb4KetwGiKHORyiZ1YQ6ip3QtzZOHYNeVqbAz54HHaNN4dLHjO69AbLFaBTKVKOgM1FLaBVoPFqKE08='} 2023-06-22T17:29:14.888 INFO:teuthology.orchestra.console:Power off smithi039 2023-06-22T17:29:14.889 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-22T17:29:14.911 INFO:teuthology.orchestra.console:Power off smithi101 2023-06-22T17:29:14.912 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi101.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-22T17:29:14.921 DEBUG:teuthology.nuke:shortname: smithi133 2023-06-22T17:29:14.922 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-22T17:29:14.927 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T17:29:14.939 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi101.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T17:29:14.953 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi133.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311823', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-22 16:58:52.439724', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkDVaRpyyMnXCswECTHN+Peaetgd+6nvXoDYKD6SLV4BaVriJBgQ4t/LC6XWMd08u74t3H63suGOlTv6BNj8S/PP1eXvqiX6XOkgH5N+ziVwNs9Ce1infLiT8/wwauHNuOBUbJYwt4BY2HmrAviuVhyRy+sH4PkvxzOAn/6YakuxoL5w8EMNX66lUfeiglR3mUKs4X1A5jNGkxfhkYfcAjXjcujgViZdDc6HhtN42INflCGfA+ukylfyo1/o8n52fEe31ClsNyrKw/TinChNWmSX8aReQKA2PiAOwk7moUOxKNQKD6lQwblKNVDs6HiOG9TOoL91azLey2f3Alsi45i2BCKOpF9YBQv1hxX/YtPCGcRyzB2SsdpSNeS8uaHXdOolCE5RB0ZKCX59mUipQYyGmiHtgI0KdV6CJJffZOwDtdGFgfYuYQxJEK9M6dJ6BezyFXz0QrUpd1M8vMiNGI4f0TYORr6wqHqCLiw0AHLUINTncfewceZDD3vbllQAU='} 2023-06-22T17:29:15.013 INFO:teuthology.orchestra.console:Power off smithi133 2023-06-22T17:29:15.014 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-22T17:29:15.039 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T17:29:18.960 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T17:29:18.971 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi101.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T17:29:19.066 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T17:29:23.080 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T17:29:23.090 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi101.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T17:29:23.180 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T17:29:23.293 INFO:teuthology.orchestra.console:Power off for smithi039 completed 2023-06-22T17:29:23.304 INFO:teuthology.orchestra.console:Power off for smithi101 completed 2023-06-22T17:29:27.295 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T17:29:27.507 INFO:teuthology.orchestra.console:Power off for smithi133 completed 2023-06-22T17:29:27.614 INFO:teuthology.run:Summary data: description: fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/pjd} wsync/{no}} duration: 1255.772494316101 failure_reason: 'Command failed on smithi039 with status 22: ''sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=abde005e46504e8cafffb0772f206a1e status: fail success: false 2023-06-22T17:29:27.615 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-22T17:29:27.708 INFO:teuthology.run:FAIL