2023-07-20T02:10:58.092 INFO:root:teuthology version: 0.0.1.dev153+g79491cb 2023-07-20T02:10:58.093 DEBUG:teuthology.run:Teuthology command: teuthology --name pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344290 --description fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsync-tester}} --verbose -- /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344290/orig.config.yaml 2023-07-20T02:10:58.163 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-20T02:10:58.272 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344290 branch: wip-pdonnell-testing-20230719.201220 description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsync-tester}} email: pdonnell@redhat.com first_in_suite: false job_id: '7344290' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-pdonnell-testing-20230719.201220 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported dry_run: true install: ceph: flavor: default sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - nowsync syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20230719.201220 sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 6483 sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 sleep_before_teardown: 0 subset: 383/512 suite: fs:workload suite_branch: wip-pdonnell-testing-20230719.201220 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi007.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= smithi158.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= smithi188.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - workunit: clients: all: - suites/fsync-tester.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 79491cb69dd37a1eafb93b3471998174bdfa2fd9 timestamp: 2023-07-20_00:39:06 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-07-20T02:10:58.272 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa; will attempt to use it 2023-07-20T02:10:58.273 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks 2023-07-20T02:10:58.274 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-07-20T02:10:58.276 INFO:teuthology.task.internal:Checking packages... 2023-07-20T02:10:58.304 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '1b0b2eb005a7c51d97a788b9900d511e1394b537' 2023-07-20T02:10:58.305 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-07-20T02:10:58.305 INFO:teuthology.packaging:ref: None 2023-07-20T02:10:58.306 INFO:teuthology.packaging:tag: None 2023-07-20T02:10:58.306 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20230719.201220 2023-07-20T02:10:58.306 INFO:teuthology.packaging:sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T02:10:58.306 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-pdonnell-testing-20230719.201220 2023-07-20T02:10:59.049 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-5040.g1b0b2eb0 2023-07-20T02:10:59.050 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-07-20T02:10:59.059 INFO:teuthology.task.internal:no buildpackages task found 2023-07-20T02:10:59.060 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-07-20T02:10:59.068 INFO:teuthology.task.internal:Saving configuration 2023-07-20T02:10:59.088 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-07-20T02:10:59.097 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-20T02:10:59.125 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344290', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 02:05:29.074186', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-20T02:10:59.152 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi158.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344290', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 02:05:29.075989', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-20T02:10:59.177 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi188.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344290', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 02:05:29.071809', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-20T02:10:59.178 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-07-20T02:10:59.185 INFO:teuthology.task.internal:roles: ubuntu@smithi007.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2023-07-20T02:10:59.185 INFO:teuthology.task.internal:roles: ubuntu@smithi158.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2023-07-20T02:10:59.185 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2023-07-20T02:10:59.185 INFO:teuthology.run_tasks:Running task console_log... 2023-07-20T02:10:59.310 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5c1bb493a0>, signals=[15]) 2023-07-20T02:10:59.311 INFO:teuthology.run_tasks:Running task internal.connect... 2023-07-20T02:10:59.319 INFO:teuthology.task.internal:Opening connections... 2023-07-20T02:10:59.319 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi007.front.sepia.ceph.com 2023-07-20T02:10:59.323 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T02:10:59.394 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi158.front.sepia.ceph.com 2023-07-20T02:10:59.395 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T02:10:59.463 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi188.front.sepia.ceph.com 2023-07-20T02:10:59.464 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T02:10:59.535 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-07-20T02:10:59.542 DEBUG:teuthology.orchestra.run.smithi007:> uname -m 2023-07-20T02:10:59.563 INFO:teuthology.orchestra.run.smithi007.stdout:x86_64 2023-07-20T02:10:59.563 DEBUG:teuthology.orchestra.run.smithi007:> cat /etc/os-release 2023-07-20T02:10:59.623 INFO:teuthology.orchestra.run.smithi007.stdout:NAME="Red Hat Enterprise Linux" 2023-07-20T02:10:59.623 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION="8.6 (Ootpa)" 2023-07-20T02:10:59.623 INFO:teuthology.orchestra.run.smithi007.stdout:ID="rhel" 2023-07-20T02:10:59.623 INFO:teuthology.orchestra.run.smithi007.stdout:ID_LIKE="fedora" 2023-07-20T02:10:59.623 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_ID="8.6" 2023-07-20T02:10:59.624 INFO:teuthology.orchestra.run.smithi007.stdout:PLATFORM_ID="platform:el8" 2023-07-20T02:10:59.624 INFO:teuthology.orchestra.run.smithi007.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-20T02:10:59.624 INFO:teuthology.orchestra.run.smithi007.stdout:ANSI_COLOR="0;31" 2023-07-20T02:10:59.624 INFO:teuthology.orchestra.run.smithi007.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-20T02:10:59.624 INFO:teuthology.orchestra.run.smithi007.stdout:HOME_URL="https://www.redhat.com/" 2023-07-20T02:10:59.624 INFO:teuthology.orchestra.run.smithi007.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-20T02:10:59.624 INFO:teuthology.orchestra.run.smithi007.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-20T02:10:59.625 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:10:59.625 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-20T02:10:59.625 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-20T02:10:59.625 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-20T02:10:59.625 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-20T02:10:59.626 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server 2023-07-20T02:10:59.650 DEBUG:teuthology.orchestra.run.smithi158:> uname -m 2023-07-20T02:10:59.670 INFO:teuthology.orchestra.run.smithi158.stdout:x86_64 2023-07-20T02:10:59.671 DEBUG:teuthology.orchestra.run.smithi158:> cat /etc/os-release 2023-07-20T02:10:59.729 INFO:teuthology.orchestra.run.smithi158.stdout:NAME="Red Hat Enterprise Linux" 2023-07-20T02:10:59.730 INFO:teuthology.orchestra.run.smithi158.stdout:VERSION="8.6 (Ootpa)" 2023-07-20T02:10:59.730 INFO:teuthology.orchestra.run.smithi158.stdout:ID="rhel" 2023-07-20T02:10:59.730 INFO:teuthology.orchestra.run.smithi158.stdout:ID_LIKE="fedora" 2023-07-20T02:10:59.730 INFO:teuthology.orchestra.run.smithi158.stdout:VERSION_ID="8.6" 2023-07-20T02:10:59.730 INFO:teuthology.orchestra.run.smithi158.stdout:PLATFORM_ID="platform:el8" 2023-07-20T02:10:59.730 INFO:teuthology.orchestra.run.smithi158.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-20T02:10:59.731 INFO:teuthology.orchestra.run.smithi158.stdout:ANSI_COLOR="0;31" 2023-07-20T02:10:59.731 INFO:teuthology.orchestra.run.smithi158.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-20T02:10:59.731 INFO:teuthology.orchestra.run.smithi158.stdout:HOME_URL="https://www.redhat.com/" 2023-07-20T02:10:59.731 INFO:teuthology.orchestra.run.smithi158.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-20T02:10:59.731 INFO:teuthology.orchestra.run.smithi158.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-20T02:10:59.731 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-07-20T02:10:59.731 INFO:teuthology.orchestra.run.smithi158.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-20T02:10:59.732 INFO:teuthology.orchestra.run.smithi158.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-20T02:10:59.732 INFO:teuthology.orchestra.run.smithi158.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-20T02:10:59.732 INFO:teuthology.orchestra.run.smithi158.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-20T02:10:59.732 INFO:teuthology.lock.ops:Updating smithi158.front.sepia.ceph.com on lock server 2023-07-20T02:10:59.757 DEBUG:teuthology.orchestra.run.smithi188:> uname -m 2023-07-20T02:10:59.778 INFO:teuthology.orchestra.run.smithi188.stdout:x86_64 2023-07-20T02:10:59.779 DEBUG:teuthology.orchestra.run.smithi188:> cat /etc/os-release 2023-07-20T02:10:59.836 INFO:teuthology.orchestra.run.smithi188.stdout:NAME="Red Hat Enterprise Linux" 2023-07-20T02:10:59.837 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION="8.6 (Ootpa)" 2023-07-20T02:10:59.837 INFO:teuthology.orchestra.run.smithi188.stdout:ID="rhel" 2023-07-20T02:10:59.837 INFO:teuthology.orchestra.run.smithi188.stdout:ID_LIKE="fedora" 2023-07-20T02:10:59.837 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION_ID="8.6" 2023-07-20T02:10:59.837 INFO:teuthology.orchestra.run.smithi188.stdout:PLATFORM_ID="platform:el8" 2023-07-20T02:10:59.837 INFO:teuthology.orchestra.run.smithi188.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-20T02:10:59.838 INFO:teuthology.orchestra.run.smithi188.stdout:ANSI_COLOR="0;31" 2023-07-20T02:10:59.838 INFO:teuthology.orchestra.run.smithi188.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-20T02:10:59.838 INFO:teuthology.orchestra.run.smithi188.stdout:HOME_URL="https://www.redhat.com/" 2023-07-20T02:10:59.838 INFO:teuthology.orchestra.run.smithi188.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-20T02:10:59.838 INFO:teuthology.orchestra.run.smithi188.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-20T02:10:59.838 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-07-20T02:10:59.838 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-20T02:10:59.839 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-20T02:10:59.839 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-20T02:10:59.839 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-20T02:10:59.839 INFO:teuthology.lock.ops:Updating smithi188.front.sepia.ceph.com on lock server 2023-07-20T02:10:59.866 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-07-20T02:10:59.881 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-07-20T02:10:59.890 INFO:teuthology.task.internal:Checking for old test directory... 2023-07-20T02:10:59.891 DEBUG:teuthology.orchestra.run.smithi007:> test '!' -e /home/ubuntu/cephtest 2023-07-20T02:10:59.896 DEBUG:teuthology.orchestra.run.smithi158:> test '!' -e /home/ubuntu/cephtest 2023-07-20T02:10:59.900 DEBUG:teuthology.orchestra.run.smithi188:> test '!' -e /home/ubuntu/cephtest 2023-07-20T02:10:59.919 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-07-20T02:10:59.926 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-07-20T02:10:59.926 DEBUG:teuthology.orchestra.run.smithi007:> test -z $(ls -A /var/lib/ceph) 2023-07-20T02:10:59.956 DEBUG:teuthology.orchestra.run.smithi158:> test -z $(ls -A /var/lib/ceph) 2023-07-20T02:10:59.960 DEBUG:teuthology.orchestra.run.smithi188:> test -z $(ls -A /var/lib/ceph) 2023-07-20T02:10:59.987 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-07-20T02:11:00.074 INFO:teuthology.run_tasks:Running task kernel... 2023-07-20T02:11:00.092 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2023-07-20T02:11:00.093 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2023-07-20T02:11:00.093 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2023-07-20T02:11:00.093 DEBUG:teuthology.orchestra.run.smithi007:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-20T02:11:00.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T02:11:00.115 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&ref=testing 2023-07-20T02:11:00.778 DEBUG:teuthology.task.kernel:sha1 for client.0 is a440c79eee0258ae07e4988d8690fe6187b8b7a3 2023-07-20T02:11:00.778 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "a440c79eee0258ae07e4988d8690fe6187b8b7a3"... 2023-07-20T02:11:00.778 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2023-07-20T02:11:00.798 INFO:teuthology.orchestra.run.smithi007.stdout:4.18.0-372.9.1.el8.x86_64 2023-07-20T02:11:00.799 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-372.9.1.el8.x86_64 vs a440c79eee0258ae07e4988d8690fe6187b8b7a3 2023-07-20T02:11:00.799 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-07-20T02:11:00.800 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum upgrade -y linux-firmware 2023-07-20T02:11:02.073 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-07-20T02:11:02.073 INFO:teuthology.orchestra.run.smithi007.stdout:Unable to read consumer identity 2023-07-20T02:11:04.068 INFO:teuthology.orchestra.run.smithi007.stdout:Copr repo for python3-asyncssh owned by ceph 5.3 kB/s | 3.5 kB 00:00 2023-07-20T02:11:07.909 INFO:teuthology.orchestra.run.smithi007.stdout:Extra Packages for Enterprise Linux 4.1 MB/s | 16 MB 00:03 2023-07-20T02:11:11.609 INFO:teuthology.orchestra.run.smithi007.stdout:lab-extras 559 kB/s | 24 kB 00:00 2023-07-20T02:11:12.312 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-07-20T02:11:12.314 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2023-07-20T02:11:12.314 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-07-20T02:11:12.435 INFO:teuthology.task.kernel:Downloading kernel a440c79eee0258ae07e4988d8690fe6187b8b7a3 on client.0... 2023-07-20T02:11:12.435 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=a440c79eee0258ae07e4988d8690fe6187b8b7a3 2023-07-20T02:11:13.129 INFO:teuthology.task.kernel:fetching, builder baseurl is https://2.chacra.ceph.com/r/kernel/testing/a440c79eee0258ae07e4988d8690fe6187b8b7a3/centos/8/flavors/default/x86_64/ 2023-07-20T02:11:13.129 DEBUG:teuthology.orchestra.run.smithi007:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.5.0_rc1_ga440c79eee02-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://2.chacra.ceph.com/r/kernel/testing/a440c79eee0258ae07e4988d8690fe6187b8b7a3/centos/8/flavors/default/x86_64/ --input-file=- 2023-07-20T02:11:13.132 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi007.front.sepia.ceph.com to complete... 2023-07-20T02:19:53.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20 02:19:53 URL:https://2.chacra.ceph.com/r/kernel/testing/a440c79eee0258ae07e4988d8690fe6187b8b7a3/centos/8/flavors/default/x86_64/kernel-6.5.0_rc1_ga440c79eee02-1.x86_64.rpm [380684764/380684764] -> "/tmp/kernel.x86_64.rpm" [1] 2023-07-20T02:19:53.880 INFO:teuthology.orchestra.run.smithi007.stderr:FINISHED --2023-07-20 02:19:53-- 2023-07-20T02:19:53.880 INFO:teuthology.orchestra.run.smithi007.stderr:Total wall clock time: 8m 41s 2023-07-20T02:19:53.880 INFO:teuthology.orchestra.run.smithi007.stderr:Downloaded: 1 files, 363M in 8m 40s (715 KB/s) 2023-07-20T02:19:53.881 INFO:teuthology.task.kernel:Installing kernel a440c79eee0258ae07e4988d8690fe6187b8b7a3 on client.0... 2023-07-20T02:19:53.882 DEBUG:teuthology.orchestra.run.smithi007:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2023-07-20T02:19:55.565 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying... ######################################## 2023-07-20T02:19:55.630 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing... ######################################## 2023-07-20T02:19:55.743 INFO:teuthology.orchestra.run.smithi007.stdout:Updating / installing... 2023-07-20T02:20:36.232 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-6.5.0_rc1_ga440c79eee02-1 ######################################## 2023-07-20T02:21:00.611 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi007.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2023-07-20T02:21:00.612 DEBUG:teuthology.orchestra.run.smithi007:> rpm -qlp /tmp/kernel.x86_64.rpm 2023-07-20T02:21:00.663 INFO:teuthology.orchestra.run.smithi007.stdout:/boot/System.map-6.5.0-rc1-ga440c79eee02 2023-07-20T02:21:00.663 INFO:teuthology.orchestra.run.smithi007.stdout:/boot/config-6.5.0-rc1-ga440c79eee02 2023-07-20T02:21:00.663 INFO:teuthology.orchestra.run.smithi007.stdout:/boot/vmlinuz-6.5.0-rc1-ga440c79eee02 2023-07-20T02:21:00.664 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02 2023-07-20T02:21:00.664 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel 2023-07-20T02:21:00.664 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/arch 2023-07-20T02:21:00.664 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/arch/x86 2023-07-20T02:21:00.664 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/arch/x86/crypto 2023-07-20T02:21:00.664 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-07-20T02:21:00.664 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-07-20T02:21:00.665 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-07-20T02:21:00.665 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-07-20T02:21:00.665 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-07-20T02:21:00.665 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-07-20T02:21:00.665 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-07-20T02:21:00.665 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/arch/x86/crypto/crc32c-intel.ko 2023-07-20T02:21:00.665 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-07-20T02:21:00.665 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-07-20T02:21:00.666 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-07-20T02:21:00.666 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-07-20T02:21:00.666 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/arch/x86/crypto/serpent-avx2.ko 2023-07-20T02:21:00.666 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-07-20T02:21:00.666 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-07-20T02:21:00.666 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-07-20T02:21:00.666 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-07-20T02:21:00.667 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-07-20T02:21:00.667 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-07-20T02:21:00.667 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-07-20T02:21:00.667 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/arch/x86/kernel 2023-07-20T02:21:00.668 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/arch/x86/kernel/cpu 2023-07-20T02:21:00.668 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/arch/x86/kernel/cpu/mce 2023-07-20T02:21:00.668 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-07-20T02:21:00.669 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/arch/x86/kvm 2023-07-20T02:21:00.669 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/arch/x86/kvm/kvm-amd.ko 2023-07-20T02:21:00.669 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/arch/x86/kvm/kvm-intel.ko 2023-07-20T02:21:00.669 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/arch/x86/kvm/kvm.ko 2023-07-20T02:21:00.669 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto 2023-07-20T02:21:00.669 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/ansi_cprng.ko 2023-07-20T02:21:00.669 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/anubis.ko 2023-07-20T02:21:00.670 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/arc4.ko 2023-07-20T02:21:00.670 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/async_tx 2023-07-20T02:21:00.670 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/async_tx/async_memcpy.ko 2023-07-20T02:21:00.670 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/async_tx/async_pq.ko 2023-07-20T02:21:00.670 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/async_tx/async_raid6_recov.ko 2023-07-20T02:21:00.670 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/async_tx/async_tx.ko 2023-07-20T02:21:00.670 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/async_tx/async_xor.ko 2023-07-20T02:21:00.670 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/async_tx/raid6test.ko 2023-07-20T02:21:00.671 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/authenc.ko 2023-07-20T02:21:00.671 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/authencesn.ko 2023-07-20T02:21:00.671 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/blake2b_generic.ko 2023-07-20T02:21:00.671 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/blowfish_common.ko 2023-07-20T02:21:00.671 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/blowfish_generic.ko 2023-07-20T02:21:00.671 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/camellia_generic.ko 2023-07-20T02:21:00.671 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/cast5_generic.ko 2023-07-20T02:21:00.672 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/cast6_generic.ko 2023-07-20T02:21:00.672 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/cast_common.ko 2023-07-20T02:21:00.672 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/ccm.ko 2023-07-20T02:21:00.672 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/cmac.ko 2023-07-20T02:21:00.672 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/crc32_generic.ko 2023-07-20T02:21:00.672 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/crypto_engine.ko 2023-07-20T02:21:00.672 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/crypto_user.ko 2023-07-20T02:21:00.672 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/cts.ko 2023-07-20T02:21:00.673 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/des_generic.ko 2023-07-20T02:21:00.673 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/dh_generic.ko 2023-07-20T02:21:00.673 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/echainiv.ko 2023-07-20T02:21:00.673 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/essiv.ko 2023-07-20T02:21:00.673 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/fcrypt.ko 2023-07-20T02:21:00.673 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/khazad.ko 2023-07-20T02:21:00.673 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/lz4.ko 2023-07-20T02:21:00.674 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/lz4hc.ko 2023-07-20T02:21:00.674 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/md4.ko 2023-07-20T02:21:00.674 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/michael_mic.ko 2023-07-20T02:21:00.674 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/pcbc.ko 2023-07-20T02:21:00.674 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/pcrypt.ko 2023-07-20T02:21:00.674 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/rmd160.ko 2023-07-20T02:21:00.674 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/seed.ko 2023-07-20T02:21:00.675 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/serpent_generic.ko 2023-07-20T02:21:00.675 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/tcrypt.ko 2023-07-20T02:21:00.675 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/tea.ko 2023-07-20T02:21:00.675 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/twofish_common.ko 2023-07-20T02:21:00.675 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/twofish_generic.ko 2023-07-20T02:21:00.675 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/vmac.ko 2023-07-20T02:21:00.675 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/wp512.ko 2023-07-20T02:21:00.675 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/xcbc.ko 2023-07-20T02:21:00.676 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/xor.ko 2023-07-20T02:21:00.676 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/crypto/xxhash_generic.ko 2023-07-20T02:21:00.676 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers 2023-07-20T02:21:00.676 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/acpi 2023-07-20T02:21:00.676 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/acpi/acpi_ipmi.ko 2023-07-20T02:21:00.676 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/acpi/acpi_pad.ko 2023-07-20T02:21:00.676 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/acpi/custom_method.ko 2023-07-20T02:21:00.676 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/acpi/ec_sys.ko 2023-07-20T02:21:00.677 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/acpi/platform_profile.ko 2023-07-20T02:21:00.677 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/acpi/sbs.ko 2023-07-20T02:21:00.677 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/acpi/sbshc.ko 2023-07-20T02:21:00.677 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/acpi/video.ko 2023-07-20T02:21:00.677 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata 2023-07-20T02:21:00.677 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/acard-ahci.ko 2023-07-20T02:21:00.677 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/ahci_platform.ko 2023-07-20T02:21:00.678 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/ata_generic.ko 2023-07-20T02:21:00.678 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/libahci_platform.ko 2023-07-20T02:21:00.678 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_acpi.ko 2023-07-20T02:21:00.678 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_ali.ko 2023-07-20T02:21:00.678 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_amd.ko 2023-07-20T02:21:00.678 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_artop.ko 2023-07-20T02:21:00.678 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_atiixp.ko 2023-07-20T02:21:00.678 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_atp867x.ko 2023-07-20T02:21:00.679 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_cmd640.ko 2023-07-20T02:21:00.679 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_cmd64x.ko 2023-07-20T02:21:00.679 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_cypress.ko 2023-07-20T02:21:00.679 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_efar.ko 2023-07-20T02:21:00.679 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_hpt366.ko 2023-07-20T02:21:00.679 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_hpt37x.ko 2023-07-20T02:21:00.679 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_hpt3x2n.ko 2023-07-20T02:21:00.680 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_hpt3x3.ko 2023-07-20T02:21:00.680 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_it8213.ko 2023-07-20T02:21:00.680 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_it821x.ko 2023-07-20T02:21:00.680 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_jmicron.ko 2023-07-20T02:21:00.680 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_marvell.ko 2023-07-20T02:21:00.680 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_mpiix.ko 2023-07-20T02:21:00.680 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_netcell.ko 2023-07-20T02:21:00.680 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_ninja32.ko 2023-07-20T02:21:00.681 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_ns87410.ko 2023-07-20T02:21:00.681 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_ns87415.ko 2023-07-20T02:21:00.681 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_oldpiix.ko 2023-07-20T02:21:00.681 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_opti.ko 2023-07-20T02:21:00.681 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_optidma.ko 2023-07-20T02:21:00.681 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_pcmcia.ko 2023-07-20T02:21:00.681 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_pdc2027x.ko 2023-07-20T02:21:00.682 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-07-20T02:21:00.682 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_piccolo.ko 2023-07-20T02:21:00.682 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_rdc.ko 2023-07-20T02:21:00.682 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_sch.ko 2023-07-20T02:21:00.682 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_serverworks.ko 2023-07-20T02:21:00.682 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_sil680.ko 2023-07-20T02:21:00.682 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_sis.ko 2023-07-20T02:21:00.682 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_sl82c105.ko 2023-07-20T02:21:00.683 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_triflex.ko 2023-07-20T02:21:00.683 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pata_via.ko 2023-07-20T02:21:00.683 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/pdc_adma.ko 2023-07-20T02:21:00.683 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/sata_inic162x.ko 2023-07-20T02:21:00.683 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/sata_mv.ko 2023-07-20T02:21:00.683 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/sata_nv.ko 2023-07-20T02:21:00.683 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/sata_promise.ko 2023-07-20T02:21:00.683 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/sata_qstor.ko 2023-07-20T02:21:00.684 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/sata_sil.ko 2023-07-20T02:21:00.684 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/sata_sil24.ko 2023-07-20T02:21:00.684 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/sata_sis.ko 2023-07-20T02:21:00.684 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/sata_svw.ko 2023-07-20T02:21:00.684 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/sata_sx4.ko 2023-07-20T02:21:00.684 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/sata_uli.ko 2023-07-20T02:21:00.684 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/sata_via.ko 2023-07-20T02:21:00.684 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ata/sata_vsc.ko 2023-07-20T02:21:00.685 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/auxdisplay 2023-07-20T02:21:00.685 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/auxdisplay/ks0108.ko 2023-07-20T02:21:00.685 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/base 2023-07-20T02:21:00.685 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/base/regmap 2023-07-20T02:21:00.685 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/base/regmap/regmap-i2c.ko 2023-07-20T02:21:00.685 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/base/regmap/regmap-mmio.ko 2023-07-20T02:21:00.685 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/bcma 2023-07-20T02:21:00.686 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/bcma/bcma.ko 2023-07-20T02:21:00.686 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/block 2023-07-20T02:21:00.686 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/block/aoe 2023-07-20T02:21:00.686 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/block/aoe/aoe.ko 2023-07-20T02:21:00.686 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/block/brd.ko 2023-07-20T02:21:00.686 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/block/drbd 2023-07-20T02:21:00.686 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/block/drbd/drbd.ko 2023-07-20T02:21:00.686 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/block/floppy.ko 2023-07-20T02:21:00.687 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/block/loop.ko 2023-07-20T02:21:00.687 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/block/mtip32xx 2023-07-20T02:21:00.687 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-07-20T02:21:00.687 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/block/nbd.ko 2023-07-20T02:21:00.687 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/block/null_blk 2023-07-20T02:21:00.687 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/block/null_blk/null_blk.ko 2023-07-20T02:21:00.687 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/block/pktcdvd.ko 2023-07-20T02:21:00.687 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/block/rbd.ko 2023-07-20T02:21:00.688 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/block/virtio_blk.ko 2023-07-20T02:21:00.688 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/block/xen-blkback 2023-07-20T02:21:00.688 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-07-20T02:21:00.688 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/block/xen-blkfront.ko 2023-07-20T02:21:00.688 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/block/zram 2023-07-20T02:21:00.688 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/block/zram/zram.ko 2023-07-20T02:21:00.688 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/char 2023-07-20T02:21:00.688 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/char/hangcheck-timer.ko 2023-07-20T02:21:00.689 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/char/hw_random 2023-07-20T02:21:00.689 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/char/hw_random/amd-rng.ko 2023-07-20T02:21:00.689 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/char/hw_random/intel-rng.ko 2023-07-20T02:21:00.689 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-07-20T02:21:00.689 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/char/hw_random/via-rng.ko 2023-07-20T02:21:00.689 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/char/hw_random/virtio-rng.ko 2023-07-20T02:21:00.689 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/char/ipmi 2023-07-20T02:21:00.689 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-07-20T02:21:00.690 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-07-20T02:21:00.690 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-07-20T02:21:00.690 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/char/ipmi/ipmi_si.ko 2023-07-20T02:21:00.690 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-07-20T02:21:00.690 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-07-20T02:21:00.690 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/char/lp.ko 2023-07-20T02:21:00.690 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/char/mwave 2023-07-20T02:21:00.691 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/char/mwave/mwave.ko 2023-07-20T02:21:00.691 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/char/ppdev.ko 2023-07-20T02:21:00.691 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/char/tlclk.ko 2023-07-20T02:21:00.691 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/char/tpm 2023-07-20T02:21:00.691 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/char/tpm/tpm.ko 2023-07-20T02:21:00.691 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/char/tpm/tpm_atmel.ko 2023-07-20T02:21:00.691 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/char/tpm/tpm_infineon.ko 2023-07-20T02:21:00.691 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/char/tpm/tpm_nsc.ko 2023-07-20T02:21:00.692 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/char/tpm/tpm_tis.ko 2023-07-20T02:21:00.692 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/char/tpm/tpm_tis_core.ko 2023-07-20T02:21:00.692 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/char/uv_mmtimer.ko 2023-07-20T02:21:00.692 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/char/virtio_console.ko 2023-07-20T02:21:00.692 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/cpufreq 2023-07-20T02:21:00.692 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/cpufreq/acpi-cpufreq.ko 2023-07-20T02:21:00.692 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-07-20T02:21:00.692 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/cpufreq/p4-clockmod.ko 2023-07-20T02:21:00.693 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/cpufreq/pcc-cpufreq.ko 2023-07-20T02:21:00.693 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/cpufreq/powernow-k8.ko 2023-07-20T02:21:00.693 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/cpufreq/speedstep-lib.ko 2023-07-20T02:21:00.693 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/crypto 2023-07-20T02:21:00.693 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/crypto/ccp 2023-07-20T02:21:00.693 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-07-20T02:21:00.693 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/crypto/ccp/ccp.ko 2023-07-20T02:21:00.694 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/crypto/intel 2023-07-20T02:21:00.694 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/crypto/intel/qat 2023-07-20T02:21:00.694 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/crypto/intel/qat/qat_common 2023-07-20T02:21:00.694 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-07-20T02:21:00.694 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2023-07-20T02:21:00.694 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-07-20T02:21:00.694 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/crypto/padlock-aes.ko 2023-07-20T02:21:00.694 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/crypto/padlock-sha.ko 2023-07-20T02:21:00.695 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/crypto/virtio 2023-07-20T02:21:00.695 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-07-20T02:21:00.695 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/dca 2023-07-20T02:21:00.695 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/dca/dca.ko 2023-07-20T02:21:00.695 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/devfreq 2023-07-20T02:21:00.695 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/devfreq/governor_simpleondemand.ko 2023-07-20T02:21:00.695 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/dma 2023-07-20T02:21:00.695 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/dma/dw 2023-07-20T02:21:00.696 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/dma/dw/dw_dmac.ko 2023-07-20T02:21:00.696 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/dma/ioat 2023-07-20T02:21:00.696 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/dma/ioat/ioatdma.ko 2023-07-20T02:21:00.696 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/edac 2023-07-20T02:21:00.696 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/edac/amd64_edac.ko 2023-07-20T02:21:00.696 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/edac/e752x_edac.ko 2023-07-20T02:21:00.696 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/edac/edac_mce_amd.ko 2023-07-20T02:21:00.697 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/edac/i3000_edac.ko 2023-07-20T02:21:00.697 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/edac/i3200_edac.ko 2023-07-20T02:21:00.697 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/edac/i5100_edac.ko 2023-07-20T02:21:00.697 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/edac/i5400_edac.ko 2023-07-20T02:21:00.697 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/edac/i7300_edac.ko 2023-07-20T02:21:00.697 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/edac/i7core_edac.ko 2023-07-20T02:21:00.697 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/edac/i82975x_edac.ko 2023-07-20T02:21:00.697 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/edac/ie31200_edac.ko 2023-07-20T02:21:00.698 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/edac/sb_edac.ko 2023-07-20T02:21:00.698 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/edac/x38_edac.ko 2023-07-20T02:21:00.698 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/firmware 2023-07-20T02:21:00.698 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/firmware/edd.ko 2023-07-20T02:21:00.698 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/firmware/iscsi_ibft.ko 2023-07-20T02:21:00.698 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/gpio 2023-07-20T02:21:00.698 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/gpio/gpio-ich.ko 2023-07-20T02:21:00.698 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/gpio/gpio-viperboard.ko 2023-07-20T02:21:00.699 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid 2023-07-20T02:21:00.699 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-appleir.ko 2023-07-20T02:21:00.699 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-aureal.ko 2023-07-20T02:21:00.699 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-axff.ko 2023-07-20T02:21:00.699 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-dr.ko 2023-07-20T02:21:00.699 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-elecom.ko 2023-07-20T02:21:00.699 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-elo.ko 2023-07-20T02:21:00.699 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-emsff.ko 2023-07-20T02:21:00.700 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-gaff.ko 2023-07-20T02:21:00.700 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-gt683r.ko 2023-07-20T02:21:00.700 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-gyration.ko 2023-07-20T02:21:00.700 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-holtek-kbd.ko 2023-07-20T02:21:00.700 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-holtek-mouse.ko 2023-07-20T02:21:00.700 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-holtekff.ko 2023-07-20T02:21:00.700 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-hyperv.ko 2023-07-20T02:21:00.700 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-icade.ko 2023-07-20T02:21:00.701 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-keytouch.ko 2023-07-20T02:21:00.701 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-kye.ko 2023-07-20T02:21:00.701 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-lcpower.ko 2023-07-20T02:21:00.701 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-led.ko 2023-07-20T02:21:00.701 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-lenovo.ko 2023-07-20T02:21:00.701 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-logitech-dj.ko 2023-07-20T02:21:00.701 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-07-20T02:21:00.702 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-multitouch.ko 2023-07-20T02:21:00.702 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-ortek.ko 2023-07-20T02:21:00.702 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-penmount.ko 2023-07-20T02:21:00.702 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-petalynx.ko 2023-07-20T02:21:00.702 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-picolcd.ko 2023-07-20T02:21:00.702 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-pl.ko 2023-07-20T02:21:00.702 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-plantronics.ko 2023-07-20T02:21:00.702 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-primax.ko 2023-07-20T02:21:00.703 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-rmi.ko 2023-07-20T02:21:00.703 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-roccat-arvo.ko 2023-07-20T02:21:00.703 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-roccat-common.ko 2023-07-20T02:21:00.703 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-roccat-isku.ko 2023-07-20T02:21:00.703 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-roccat-kone.ko 2023-07-20T02:21:00.703 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-07-20T02:21:00.703 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-roccat-konepure.ko 2023-07-20T02:21:00.703 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-07-20T02:21:00.704 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-roccat-lua.ko 2023-07-20T02:21:00.704 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-roccat-pyra.ko 2023-07-20T02:21:00.704 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-roccat-ryos.ko 2023-07-20T02:21:00.704 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-roccat-savu.ko 2023-07-20T02:21:00.704 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-roccat.ko 2023-07-20T02:21:00.704 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-saitek.ko 2023-07-20T02:21:00.704 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-samsung.ko 2023-07-20T02:21:00.705 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-sensor-hub.ko 2023-07-20T02:21:00.705 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-sjoy.ko 2023-07-20T02:21:00.705 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-sony.ko 2023-07-20T02:21:00.705 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-speedlink.ko 2023-07-20T02:21:00.705 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-steelseries.ko 2023-07-20T02:21:00.705 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-sunplus.ko 2023-07-20T02:21:00.705 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-thrustmaster.ko 2023-07-20T02:21:00.705 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-tivo.ko 2023-07-20T02:21:00.706 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-tmff.ko 2023-07-20T02:21:00.706 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-topseed.ko 2023-07-20T02:21:00.706 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-twinhan.ko 2023-07-20T02:21:00.706 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-uclogic.ko 2023-07-20T02:21:00.706 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-waltop.ko 2023-07-20T02:21:00.706 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-wiimote.ko 2023-07-20T02:21:00.706 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-xinmo.ko 2023-07-20T02:21:00.706 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-zpff.ko 2023-07-20T02:21:00.707 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/hid-zydacron.ko 2023-07-20T02:21:00.707 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/uhid.ko 2023-07-20T02:21:00.707 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hid/wacom.ko 2023-07-20T02:21:00.707 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hv 2023-07-20T02:21:00.707 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hv/hv_balloon.ko 2023-07-20T02:21:00.707 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hv/hv_utils.ko 2023-07-20T02:21:00.707 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hv/hv_vmbus.ko 2023-07-20T02:21:00.707 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon 2023-07-20T02:21:00.708 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/abituguru.ko 2023-07-20T02:21:00.708 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/abituguru3.ko 2023-07-20T02:21:00.708 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/acpi_power_meter.ko 2023-07-20T02:21:00.708 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/ad7414.ko 2023-07-20T02:21:00.708 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/ad7418.ko 2023-07-20T02:21:00.708 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/adc128d818.ko 2023-07-20T02:21:00.708 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/adm1025.ko 2023-07-20T02:21:00.709 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/adm1026.ko 2023-07-20T02:21:00.709 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/adm1029.ko 2023-07-20T02:21:00.709 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/adm1031.ko 2023-07-20T02:21:00.709 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/adm9240.ko 2023-07-20T02:21:00.709 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/ads7828.ko 2023-07-20T02:21:00.709 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/adt7410.ko 2023-07-20T02:21:00.709 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/adt7411.ko 2023-07-20T02:21:00.709 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/adt7462.ko 2023-07-20T02:21:00.710 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/adt7470.ko 2023-07-20T02:21:00.710 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/adt7475.ko 2023-07-20T02:21:00.710 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/adt7x10.ko 2023-07-20T02:21:00.710 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/amc6821.ko 2023-07-20T02:21:00.710 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/applesmc.ko 2023-07-20T02:21:00.710 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/asb100.ko 2023-07-20T02:21:00.710 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/asc7621.ko 2023-07-20T02:21:00.710 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/asus_atk0110.ko 2023-07-20T02:21:00.711 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/atxp1.ko 2023-07-20T02:21:00.711 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/coretemp.ko 2023-07-20T02:21:00.711 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-07-20T02:21:00.711 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/dme1737.ko 2023-07-20T02:21:00.711 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/ds1621.ko 2023-07-20T02:21:00.711 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/ds620.ko 2023-07-20T02:21:00.711 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/emc1403.ko 2023-07-20T02:21:00.712 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/emc6w201.ko 2023-07-20T02:21:00.712 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/f71805f.ko 2023-07-20T02:21:00.712 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/f71882fg.ko 2023-07-20T02:21:00.712 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/f75375s.ko 2023-07-20T02:21:00.712 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/fam15h_power.ko 2023-07-20T02:21:00.712 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/fschmd.ko 2023-07-20T02:21:00.712 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/g760a.ko 2023-07-20T02:21:00.713 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/g762.ko 2023-07-20T02:21:00.713 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/gl518sm.ko 2023-07-20T02:21:00.713 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/gl520sm.ko 2023-07-20T02:21:00.713 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/hwmon-vid.ko 2023-07-20T02:21:00.713 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/i5500_temp.ko 2023-07-20T02:21:00.713 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/i5k_amb.ko 2023-07-20T02:21:00.713 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/ibmaem.ko 2023-07-20T02:21:00.713 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/ibmpex.ko 2023-07-20T02:21:00.714 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/ina209.ko 2023-07-20T02:21:00.714 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/ina2xx.ko 2023-07-20T02:21:00.714 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/it87.ko 2023-07-20T02:21:00.714 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/k10temp.ko 2023-07-20T02:21:00.714 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/k8temp.ko 2023-07-20T02:21:00.714 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/lineage-pem.ko 2023-07-20T02:21:00.714 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/lm63.ko 2023-07-20T02:21:00.714 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/lm73.ko 2023-07-20T02:21:00.715 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/lm75.ko 2023-07-20T02:21:00.715 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/lm77.ko 2023-07-20T02:21:00.715 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/lm78.ko 2023-07-20T02:21:00.715 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/lm80.ko 2023-07-20T02:21:00.715 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/lm83.ko 2023-07-20T02:21:00.715 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/lm85.ko 2023-07-20T02:21:00.715 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/lm87.ko 2023-07-20T02:21:00.715 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/lm90.ko 2023-07-20T02:21:00.716 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/lm92.ko 2023-07-20T02:21:00.716 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/lm93.ko 2023-07-20T02:21:00.716 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/lm95234.ko 2023-07-20T02:21:00.716 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/lm95241.ko 2023-07-20T02:21:00.716 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/lm95245.ko 2023-07-20T02:21:00.716 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/ltc2945.ko 2023-07-20T02:21:00.716 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/ltc4151.ko 2023-07-20T02:21:00.717 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/ltc4215.ko 2023-07-20T02:21:00.717 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/ltc4222.ko 2023-07-20T02:21:00.717 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/ltc4245.ko 2023-07-20T02:21:00.717 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/ltc4260.ko 2023-07-20T02:21:00.717 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/ltc4261.ko 2023-07-20T02:21:00.717 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/max16065.ko 2023-07-20T02:21:00.717 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/max1619.ko 2023-07-20T02:21:00.717 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/max1668.ko 2023-07-20T02:21:00.718 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/max197.ko 2023-07-20T02:21:00.718 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/max6639.ko 2023-07-20T02:21:00.718 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/max6650.ko 2023-07-20T02:21:00.718 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/max6697.ko 2023-07-20T02:21:00.718 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/mcp3021.ko 2023-07-20T02:21:00.718 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/nct6683.ko 2023-07-20T02:21:00.718 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/nct6775-core.ko 2023-07-20T02:21:00.718 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/nct6775.ko 2023-07-20T02:21:00.719 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/nct7802.ko 2023-07-20T02:21:00.719 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/pc87360.ko 2023-07-20T02:21:00.719 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/pc87427.ko 2023-07-20T02:21:00.719 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/pcf8591.ko 2023-07-20T02:21:00.719 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/pmbus 2023-07-20T02:21:00.719 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-07-20T02:21:00.719 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-07-20T02:21:00.719 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-07-20T02:21:00.720 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/pmbus/max16064.ko 2023-07-20T02:21:00.720 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/pmbus/max34440.ko 2023-07-20T02:21:00.720 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/pmbus/max8688.ko 2023-07-20T02:21:00.720 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-07-20T02:21:00.720 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-07-20T02:21:00.720 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-07-20T02:21:00.720 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-07-20T02:21:00.720 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-07-20T02:21:00.721 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-07-20T02:21:00.721 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/powr1220.ko 2023-07-20T02:21:00.721 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/sch5627.ko 2023-07-20T02:21:00.721 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/sch5636.ko 2023-07-20T02:21:00.721 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/sch56xx-common.ko 2023-07-20T02:21:00.721 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/sht15.ko 2023-07-20T02:21:00.721 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/sht21.ko 2023-07-20T02:21:00.721 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/shtc1.ko 2023-07-20T02:21:00.722 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/sis5595.ko 2023-07-20T02:21:00.722 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/smsc47b397.ko 2023-07-20T02:21:00.722 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/smsc47m1.ko 2023-07-20T02:21:00.722 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/smsc47m192.ko 2023-07-20T02:21:00.722 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/thmc50.ko 2023-07-20T02:21:00.722 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/tmp102.ko 2023-07-20T02:21:00.722 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/tmp103.ko 2023-07-20T02:21:00.723 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/tmp401.ko 2023-07-20T02:21:00.723 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/tmp421.ko 2023-07-20T02:21:00.723 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/via-cputemp.ko 2023-07-20T02:21:00.723 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/via686a.ko 2023-07-20T02:21:00.723 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/vt1211.ko 2023-07-20T02:21:00.723 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/vt8231.ko 2023-07-20T02:21:00.723 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/w83627ehf.ko 2023-07-20T02:21:00.723 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/w83627hf.ko 2023-07-20T02:21:00.724 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/w83781d.ko 2023-07-20T02:21:00.724 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/w83791d.ko 2023-07-20T02:21:00.724 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/w83792d.ko 2023-07-20T02:21:00.724 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/w83793.ko 2023-07-20T02:21:00.724 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/w83795.ko 2023-07-20T02:21:00.724 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/w83l785ts.ko 2023-07-20T02:21:00.724 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/hwmon/w83l786ng.ko 2023-07-20T02:21:00.724 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/i2c 2023-07-20T02:21:00.725 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/i2c/algos 2023-07-20T02:21:00.725 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-07-20T02:21:00.725 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-07-20T02:21:00.725 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/i2c/busses 2023-07-20T02:21:00.725 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-07-20T02:21:00.725 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-07-20T02:21:00.725 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-07-20T02:21:00.725 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-07-20T02:21:00.726 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-07-20T02:21:00.726 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-07-20T02:21:00.726 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-07-20T02:21:00.726 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-07-20T02:21:00.726 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/i2c/busses/i2c-i801.ko 2023-07-20T02:21:00.726 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/i2c/busses/i2c-isch.ko 2023-07-20T02:21:00.726 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-07-20T02:21:00.726 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-07-20T02:21:00.727 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-07-20T02:21:00.727 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/i2c/busses/i2c-parport.ko 2023-07-20T02:21:00.727 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-07-20T02:21:00.727 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-07-20T02:21:00.727 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-07-20T02:21:00.727 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-07-20T02:21:00.727 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-07-20T02:21:00.728 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-07-20T02:21:00.728 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/i2c/busses/i2c-via.ko 2023-07-20T02:21:00.728 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-07-20T02:21:00.728 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-07-20T02:21:00.728 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/i2c/i2c-dev.ko 2023-07-20T02:21:00.728 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/i2c/i2c-mux.ko 2023-07-20T02:21:00.728 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/i2c/i2c-slave-eeprom.ko 2023-07-20T02:21:00.728 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/i2c/i2c-smbus.ko 2023-07-20T02:21:00.729 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/i2c/i2c-stub.ko 2023-07-20T02:21:00.729 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/infiniband 2023-07-20T02:21:00.729 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/infiniband/core 2023-07-20T02:21:00.729 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/infiniband/core/ib_cm.ko 2023-07-20T02:21:00.729 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/infiniband/core/ib_core.ko 2023-07-20T02:21:00.729 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/infiniband/core/ib_umad.ko 2023-07-20T02:21:00.729 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-07-20T02:21:00.729 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/infiniband/core/iw_cm.ko 2023-07-20T02:21:00.730 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/infiniband/core/rdma_cm.ko 2023-07-20T02:21:00.730 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-07-20T02:21:00.730 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/infiniband/hw 2023-07-20T02:21:00.730 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/infiniband/hw/cxgb4 2023-07-20T02:21:00.730 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-07-20T02:21:00.730 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/infiniband/hw/mlx4 2023-07-20T02:21:00.730 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-07-20T02:21:00.730 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/infiniband/hw/mlx5 2023-07-20T02:21:00.731 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-07-20T02:21:00.731 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/infiniband/hw/mthca 2023-07-20T02:21:00.731 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-07-20T02:21:00.731 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/infiniband/ulp 2023-07-20T02:21:00.731 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/infiniband/ulp/ipoib 2023-07-20T02:21:00.731 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-07-20T02:21:00.731 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/infiniband/ulp/iser 2023-07-20T02:21:00.731 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-07-20T02:21:00.732 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/infiniband/ulp/isert 2023-07-20T02:21:00.732 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-07-20T02:21:00.732 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/infiniband/ulp/srp 2023-07-20T02:21:00.732 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-07-20T02:21:00.732 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/infiniband/ulp/srpt 2023-07-20T02:21:00.732 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-07-20T02:21:00.732 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/input 2023-07-20T02:21:00.733 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/input/gameport 2023-07-20T02:21:00.733 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-07-20T02:21:00.733 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/input/gameport/fm801-gp.ko 2023-07-20T02:21:00.733 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/input/gameport/gameport.ko 2023-07-20T02:21:00.733 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/input/gameport/lightning.ko 2023-07-20T02:21:00.733 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/input/gameport/ns558.ko 2023-07-20T02:21:00.733 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/input/joydev.ko 2023-07-20T02:21:00.733 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/input/keyboard 2023-07-20T02:21:00.734 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/input/keyboard/gpio_keys.ko 2023-07-20T02:21:00.734 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/input/mouse 2023-07-20T02:21:00.734 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/input/mouse/appletouch.ko 2023-07-20T02:21:00.734 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/input/mouse/bcm5974.ko 2023-07-20T02:21:00.734 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/input/mouse/cyapatp.ko 2023-07-20T02:21:00.734 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/input/mouse/elan_i2c.ko 2023-07-20T02:21:00.734 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/input/mouse/sermouse.ko 2023-07-20T02:21:00.734 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-07-20T02:21:00.735 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/input/mouse/synaptics_usb.ko 2023-07-20T02:21:00.735 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/input/mouse/vsxxxaa.ko 2023-07-20T02:21:00.735 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/input/rmi4 2023-07-20T02:21:00.735 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/input/rmi4/rmi_core.ko 2023-07-20T02:21:00.735 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/input/serio 2023-07-20T02:21:00.735 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/input/serio/altera_ps2.ko 2023-07-20T02:21:00.735 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/input/serio/arc_ps2.ko 2023-07-20T02:21:00.735 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-07-20T02:21:00.736 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/input/serio/serio_raw.ko 2023-07-20T02:21:00.736 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/input/sparse-keymap.ko 2023-07-20T02:21:00.736 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/iommu 2023-07-20T02:21:00.736 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/iommu/amd 2023-07-20T02:21:00.736 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/iommu/amd/iommu_v2.ko 2023-07-20T02:21:00.736 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/leds 2023-07-20T02:21:00.736 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/leds/leds-blinkm.ko 2023-07-20T02:21:00.737 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/leds/leds-lm3530.ko 2023-07-20T02:21:00.737 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/leds/leds-lp3944.ko 2023-07-20T02:21:00.737 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/leds/leds-ss4200.ko 2023-07-20T02:21:00.737 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/leds/trigger 2023-07-20T02:21:00.737 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-07-20T02:21:00.737 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-07-20T02:21:00.737 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-07-20T02:21:00.737 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-07-20T02:21:00.738 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-07-20T02:21:00.738 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-07-20T02:21:00.738 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-07-20T02:21:00.738 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-07-20T02:21:00.738 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/md 2023-07-20T02:21:00.738 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/md/bcache 2023-07-20T02:21:00.738 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/md/bcache/bcache.ko 2023-07-20T02:21:00.738 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/md/dm-bio-prison.ko 2023-07-20T02:21:00.739 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/md/dm-cache-smq.ko 2023-07-20T02:21:00.739 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/md/dm-cache.ko 2023-07-20T02:21:00.739 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/md/dm-crypt.ko 2023-07-20T02:21:00.739 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/md/dm-delay.ko 2023-07-20T02:21:00.739 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/md/dm-flakey.ko 2023-07-20T02:21:00.739 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/md/dm-log-userspace.ko 2023-07-20T02:21:00.739 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/md/dm-multipath.ko 2023-07-20T02:21:00.739 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/md/dm-queue-length.ko 2023-07-20T02:21:00.740 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/md/dm-raid.ko 2023-07-20T02:21:00.740 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/md/dm-round-robin.ko 2023-07-20T02:21:00.740 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/md/dm-service-time.ko 2023-07-20T02:21:00.740 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/md/dm-switch.ko 2023-07-20T02:21:00.740 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/md/dm-thin-pool.ko 2023-07-20T02:21:00.740 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/md/dm-verity.ko 2023-07-20T02:21:00.740 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/md/faulty.ko 2023-07-20T02:21:00.741 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/md/linear.ko 2023-07-20T02:21:00.741 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/md/multipath.ko 2023-07-20T02:21:00.741 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/md/persistent-data 2023-07-20T02:21:00.741 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-07-20T02:21:00.741 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/md/raid0.ko 2023-07-20T02:21:00.741 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/md/raid1.ko 2023-07-20T02:21:00.741 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/md/raid10.ko 2023-07-20T02:21:00.741 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/md/raid456.ko 2023-07-20T02:21:00.742 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/message 2023-07-20T02:21:00.742 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/message/fusion 2023-07-20T02:21:00.742 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/message/fusion/mptbase.ko 2023-07-20T02:21:00.742 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/message/fusion/mptctl.ko 2023-07-20T02:21:00.742 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/message/fusion/mptfc.ko 2023-07-20T02:21:00.742 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/message/fusion/mptlan.ko 2023-07-20T02:21:00.742 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/message/fusion/mptsas.ko 2023-07-20T02:21:00.743 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/message/fusion/mptscsih.ko 2023-07-20T02:21:00.743 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/message/fusion/mptspi.ko 2023-07-20T02:21:00.743 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/mfd 2023-07-20T02:21:00.743 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/mfd/lpc_ich.ko 2023-07-20T02:21:00.743 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/mfd/lpc_sch.ko 2023-07-20T02:21:00.743 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/mfd/mfd-core.ko 2023-07-20T02:21:00.743 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/mfd/sm501.ko 2023-07-20T02:21:00.743 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/mfd/viperboard.ko 2023-07-20T02:21:00.744 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/mfd/vx855.ko 2023-07-20T02:21:00.744 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/mfd/wl1273-core.ko 2023-07-20T02:21:00.744 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc 2023-07-20T02:21:00.744 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/altera-stapl 2023-07-20T02:21:00.744 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-07-20T02:21:00.744 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/apds9802als.ko 2023-07-20T02:21:00.744 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/apds990x.ko 2023-07-20T02:21:00.744 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/bh1770glc.ko 2023-07-20T02:21:00.745 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/cb710 2023-07-20T02:21:00.745 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/cb710/cb710.ko 2023-07-20T02:21:00.745 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/eeprom 2023-07-20T02:21:00.745 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/eeprom/at24.ko 2023-07-20T02:21:00.745 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/eeprom/eeprom.ko 2023-07-20T02:21:00.745 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-07-20T02:21:00.745 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/eeprom/max6875.ko 2023-07-20T02:21:00.746 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/enclosure.ko 2023-07-20T02:21:00.746 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/hpilo.ko 2023-07-20T02:21:00.746 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/isl29003.ko 2023-07-20T02:21:00.746 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/isl29020.ko 2023-07-20T02:21:00.746 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/lis3lv02d 2023-07-20T02:21:00.746 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-07-20T02:21:00.746 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-07-20T02:21:00.746 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/mei 2023-07-20T02:21:00.747 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/mei/mei-me.ko 2023-07-20T02:21:00.747 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/mei/mei-txe.ko 2023-07-20T02:21:00.747 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/mei/mei.ko 2023-07-20T02:21:00.747 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/sgi-gru 2023-07-20T02:21:00.747 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/sgi-gru/gru.ko 2023-07-20T02:21:00.747 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/sgi-xp 2023-07-20T02:21:00.747 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/sgi-xp/xp.ko 2023-07-20T02:21:00.747 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/sgi-xp/xpc.ko 2023-07-20T02:21:00.748 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/sgi-xp/xpnet.ko 2023-07-20T02:21:00.748 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/tifm_7xx1.ko 2023-07-20T02:21:00.748 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/tifm_core.ko 2023-07-20T02:21:00.748 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/tsl2550.ko 2023-07-20T02:21:00.748 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/vmw_balloon.ko 2023-07-20T02:21:00.748 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/vmw_vmci 2023-07-20T02:21:00.748 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-07-20T02:21:00.748 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net 2023-07-20T02:21:00.749 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/bonding 2023-07-20T02:21:00.749 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/bonding/bonding.ko 2023-07-20T02:21:00.749 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/dsa 2023-07-20T02:21:00.749 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/dsa/b53 2023-07-20T02:21:00.749 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/dsa/b53/b53_common.ko 2023-07-20T02:21:00.749 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/dsa/bcm-sf2.ko 2023-07-20T02:21:00.749 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/dsa/mv88e6060.ko 2023-07-20T02:21:00.750 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/dsa/mv88e6xxx 2023-07-20T02:21:00.750 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-07-20T02:21:00.750 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/dummy.ko 2023-07-20T02:21:00.750 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/eql.ko 2023-07-20T02:21:00.750 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet 2023-07-20T02:21:00.750 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/3com 2023-07-20T02:21:00.750 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-07-20T02:21:00.750 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-07-20T02:21:00.751 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-07-20T02:21:00.751 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-07-20T02:21:00.751 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/8390 2023-07-20T02:21:00.751 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/8390/8390.ko 2023-07-20T02:21:00.751 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-07-20T02:21:00.751 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-07-20T02:21:00.751 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-07-20T02:21:00.751 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/adaptec 2023-07-20T02:21:00.752 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-07-20T02:21:00.752 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/agere 2023-07-20T02:21:00.752 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/agere/et131x.ko 2023-07-20T02:21:00.752 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/alteon 2023-07-20T02:21:00.752 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-07-20T02:21:00.752 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/altera 2023-07-20T02:21:00.752 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-07-20T02:21:00.752 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/amd 2023-07-20T02:21:00.753 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-07-20T02:21:00.753 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-07-20T02:21:00.753 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-07-20T02:21:00.753 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/atheros 2023-07-20T02:21:00.753 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/atheros/alx 2023-07-20T02:21:00.753 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-07-20T02:21:00.753 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/atheros/atl1c 2023-07-20T02:21:00.753 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-07-20T02:21:00.754 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/atheros/atl1e 2023-07-20T02:21:00.754 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-07-20T02:21:00.754 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/atheros/atlx 2023-07-20T02:21:00.754 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-07-20T02:21:00.754 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-07-20T02:21:00.754 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/broadcom 2023-07-20T02:21:00.754 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-07-20T02:21:00.755 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-07-20T02:21:00.755 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/broadcom/bnx2x 2023-07-20T02:21:00.755 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-07-20T02:21:00.755 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-07-20T02:21:00.755 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/broadcom/genet 2023-07-20T02:21:00.755 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-07-20T02:21:00.755 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-07-20T02:21:00.755 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/brocade 2023-07-20T02:21:00.756 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/brocade/bna 2023-07-20T02:21:00.756 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-07-20T02:21:00.756 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/chelsio 2023-07-20T02:21:00.756 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/chelsio/cxgb 2023-07-20T02:21:00.756 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-07-20T02:21:00.756 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/chelsio/cxgb3 2023-07-20T02:21:00.756 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-07-20T02:21:00.756 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/chelsio/cxgb4 2023-07-20T02:21:00.757 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-07-20T02:21:00.757 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2023-07-20T02:21:00.757 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-07-20T02:21:00.757 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/chelsio/libcxgb 2023-07-20T02:21:00.757 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-07-20T02:21:00.757 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/cisco 2023-07-20T02:21:00.757 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/cisco/enic 2023-07-20T02:21:00.758 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-07-20T02:21:00.758 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/dec 2023-07-20T02:21:00.758 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/dec/tulip 2023-07-20T02:21:00.758 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-07-20T02:21:00.758 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-07-20T02:21:00.758 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-07-20T02:21:00.758 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-07-20T02:21:00.758 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-07-20T02:21:00.759 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-07-20T02:21:00.759 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/dlink 2023-07-20T02:21:00.759 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-07-20T02:21:00.759 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-07-20T02:21:00.759 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/dnet.ko 2023-07-20T02:21:00.759 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/emulex 2023-07-20T02:21:00.759 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/emulex/benet 2023-07-20T02:21:00.759 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-07-20T02:21:00.760 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/ethoc.ko 2023-07-20T02:21:00.760 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/fealnx.ko 2023-07-20T02:21:00.760 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/intel 2023-07-20T02:21:00.760 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/intel/e100.ko 2023-07-20T02:21:00.760 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/intel/e1000 2023-07-20T02:21:00.760 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-07-20T02:21:00.760 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/intel/e1000e 2023-07-20T02:21:00.760 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-07-20T02:21:00.761 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/intel/fm10k 2023-07-20T02:21:00.761 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-07-20T02:21:00.761 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/intel/i40e 2023-07-20T02:21:00.761 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-07-20T02:21:00.761 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/intel/iavf 2023-07-20T02:21:00.761 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-07-20T02:21:00.761 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/intel/igb 2023-07-20T02:21:00.761 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-07-20T02:21:00.762 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/intel/igbvf 2023-07-20T02:21:00.762 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-07-20T02:21:00.762 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/intel/ixgbe 2023-07-20T02:21:00.762 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-07-20T02:21:00.762 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/intel/ixgbevf 2023-07-20T02:21:00.762 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-07-20T02:21:00.762 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/jme.ko 2023-07-20T02:21:00.762 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/marvell 2023-07-20T02:21:00.763 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-07-20T02:21:00.763 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/marvell/skge.ko 2023-07-20T02:21:00.763 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-07-20T02:21:00.763 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/mellanox 2023-07-20T02:21:00.763 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/mellanox/mlx4 2023-07-20T02:21:00.763 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-07-20T02:21:00.763 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-07-20T02:21:00.763 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/mellanox/mlx5 2023-07-20T02:21:00.764 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/mellanox/mlx5/core 2023-07-20T02:21:00.764 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-07-20T02:21:00.764 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/micrel 2023-07-20T02:21:00.764 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-07-20T02:21:00.764 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/myricom 2023-07-20T02:21:00.764 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/myricom/myri10ge 2023-07-20T02:21:00.764 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-07-20T02:21:00.765 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/natsemi 2023-07-20T02:21:00.765 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-07-20T02:21:00.765 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-07-20T02:21:00.765 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/neterion 2023-07-20T02:21:00.765 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-07-20T02:21:00.765 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/nvidia 2023-07-20T02:21:00.765 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-07-20T02:21:00.765 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/packetengines 2023-07-20T02:21:00.766 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-07-20T02:21:00.766 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-07-20T02:21:00.766 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/qlogic 2023-07-20T02:21:00.766 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/qlogic/netxen 2023-07-20T02:21:00.766 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-07-20T02:21:00.766 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-07-20T02:21:00.766 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/qlogic/qlcnic 2023-07-20T02:21:00.766 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-07-20T02:21:00.767 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/rdc 2023-07-20T02:21:00.767 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-07-20T02:21:00.767 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/realtek 2023-07-20T02:21:00.767 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-07-20T02:21:00.767 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-07-20T02:21:00.767 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/realtek/atp.ko 2023-07-20T02:21:00.767 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-07-20T02:21:00.767 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/rocker 2023-07-20T02:21:00.768 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/rocker/rocker.ko 2023-07-20T02:21:00.768 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/sfc 2023-07-20T02:21:00.768 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-07-20T02:21:00.768 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/silan 2023-07-20T02:21:00.768 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-07-20T02:21:00.768 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/sis 2023-07-20T02:21:00.768 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/sis/sis190.ko 2023-07-20T02:21:00.768 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/sis/sis900.ko 2023-07-20T02:21:00.769 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/smsc 2023-07-20T02:21:00.769 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-07-20T02:21:00.769 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-07-20T02:21:00.769 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-07-20T02:21:00.769 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-07-20T02:21:00.769 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/stmicro 2023-07-20T02:21:00.769 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/stmicro/stmmac 2023-07-20T02:21:00.769 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-07-20T02:21:00.770 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-07-20T02:21:00.770 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/sun 2023-07-20T02:21:00.770 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/sun/cassini.ko 2023-07-20T02:21:00.770 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/sun/niu.ko 2023-07-20T02:21:00.770 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/sun/sungem.ko 2023-07-20T02:21:00.770 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-07-20T02:21:00.770 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/tehuti 2023-07-20T02:21:00.771 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-07-20T02:21:00.771 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/ti 2023-07-20T02:21:00.771 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/ti/tlan.ko 2023-07-20T02:21:00.771 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/via 2023-07-20T02:21:00.771 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-07-20T02:21:00.771 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-07-20T02:21:00.771 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/wiznet 2023-07-20T02:21:00.771 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-07-20T02:21:00.772 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-07-20T02:21:00.772 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/xircom 2023-07-20T02:21:00.772 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-07-20T02:21:00.772 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/geneve.ko 2023-07-20T02:21:00.772 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/hyperv 2023-07-20T02:21:00.772 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-07-20T02:21:00.772 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ieee802154 2023-07-20T02:21:00.772 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ieee802154/fakelb.ko 2023-07-20T02:21:00.773 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ifb.ko 2023-07-20T02:21:00.773 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ipvlan 2023-07-20T02:21:00.773 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ipvlan/ipvlan.ko 2023-07-20T02:21:00.773 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/macvlan.ko 2023-07-20T02:21:00.773 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/macvtap.ko 2023-07-20T02:21:00.773 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/mdio 2023-07-20T02:21:00.773 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/mdio.ko 2023-07-20T02:21:00.773 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-07-20T02:21:00.774 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-07-20T02:21:00.774 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/mdio/mdio-regmap.ko 2023-07-20T02:21:00.774 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/mii.ko 2023-07-20T02:21:00.774 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/net_failover.ko 2023-07-20T02:21:00.774 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/netconsole.ko 2023-07-20T02:21:00.774 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/nlmon.ko 2023-07-20T02:21:00.774 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/pcs 2023-07-20T02:21:00.774 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/pcs/pcs-lynx.ko 2023-07-20T02:21:00.775 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-07-20T02:21:00.775 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/phy 2023-07-20T02:21:00.775 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/phy/amd.ko 2023-07-20T02:21:00.775 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-07-20T02:21:00.775 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/phy/bcm-phy-ptp.ko 2023-07-20T02:21:00.775 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/phy/bcm7xxx.ko 2023-07-20T02:21:00.775 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/phy/bcm87xx.ko 2023-07-20T02:21:00.775 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/phy/broadcom.ko 2023-07-20T02:21:00.776 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/phy/cicada.ko 2023-07-20T02:21:00.776 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/phy/davicom.ko 2023-07-20T02:21:00.776 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/phy/dp83640.ko 2023-07-20T02:21:00.776 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/phy/et1011c.ko 2023-07-20T02:21:00.776 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/phy/icplus.ko 2023-07-20T02:21:00.776 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/phy/lxt.ko 2023-07-20T02:21:00.776 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/phy/marvell.ko 2023-07-20T02:21:00.777 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/phy/micrel.ko 2023-07-20T02:21:00.777 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/phy/national.ko 2023-07-20T02:21:00.777 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/phy/phylink.ko 2023-07-20T02:21:00.777 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/phy/qsemi.ko 2023-07-20T02:21:00.777 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/phy/realtek.ko 2023-07-20T02:21:00.777 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/phy/smsc.ko 2023-07-20T02:21:00.777 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/phy/ste10Xp.ko 2023-07-20T02:21:00.777 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/phy/vitesse.ko 2023-07-20T02:21:00.778 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ppp 2023-07-20T02:21:00.778 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ppp/bsd_comp.ko 2023-07-20T02:21:00.778 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ppp/ppp_async.ko 2023-07-20T02:21:00.778 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ppp/ppp_deflate.ko 2023-07-20T02:21:00.778 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ppp/ppp_generic.ko 2023-07-20T02:21:00.778 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ppp/ppp_mppe.ko 2023-07-20T02:21:00.778 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ppp/ppp_synctty.ko 2023-07-20T02:21:00.778 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ppp/pppoe.ko 2023-07-20T02:21:00.779 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ppp/pppox.ko 2023-07-20T02:21:00.779 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/ppp/pptp.ko 2023-07-20T02:21:00.779 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/slip 2023-07-20T02:21:00.779 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/slip/slhc.ko 2023-07-20T02:21:00.779 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/slip/slip.ko 2023-07-20T02:21:00.779 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/sungem_phy.ko 2023-07-20T02:21:00.779 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/tap.ko 2023-07-20T02:21:00.779 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/team 2023-07-20T02:21:00.780 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/team/team.ko 2023-07-20T02:21:00.780 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/team/team_mode_activebackup.ko 2023-07-20T02:21:00.780 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/team/team_mode_broadcast.ko 2023-07-20T02:21:00.780 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-07-20T02:21:00.780 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/team/team_mode_random.ko 2023-07-20T02:21:00.780 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-07-20T02:21:00.780 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/tun.ko 2023-07-20T02:21:00.781 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/veth.ko 2023-07-20T02:21:00.781 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/virtio_net.ko 2023-07-20T02:21:00.781 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/vmxnet3 2023-07-20T02:21:00.781 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-07-20T02:21:00.781 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/vxlan 2023-07-20T02:21:00.781 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/vxlan/vxlan.ko 2023-07-20T02:21:00.781 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/xen-netback 2023-07-20T02:21:00.781 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/xen-netback/xen-netback.ko 2023-07-20T02:21:00.782 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/net/xen-netfront.ko 2023-07-20T02:21:00.782 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ntb 2023-07-20T02:21:00.782 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ntb/ntb.ko 2023-07-20T02:21:00.782 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/nvme 2023-07-20T02:21:00.782 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/nvme/host 2023-07-20T02:21:00.782 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/nvme/host/nvme-core.ko 2023-07-20T02:21:00.782 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/nvme/host/nvme.ko 2023-07-20T02:21:00.782 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/parport 2023-07-20T02:21:00.783 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/parport/parport.ko 2023-07-20T02:21:00.783 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/parport/parport_cs.ko 2023-07-20T02:21:00.783 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/parport/parport_pc.ko 2023-07-20T02:21:00.783 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/parport/parport_serial.ko 2023-07-20T02:21:00.783 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/pci 2023-07-20T02:21:00.783 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/pci/hotplug 2023-07-20T02:21:00.783 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-07-20T02:21:00.783 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/pci/pcie 2023-07-20T02:21:00.784 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/pci/pcie/aer_inject.ko 2023-07-20T02:21:00.784 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/pci/xen-pcifront.ko 2023-07-20T02:21:00.784 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/pcmcia 2023-07-20T02:21:00.784 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/pcmcia/i82092.ko 2023-07-20T02:21:00.784 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/pcmcia/pd6729.ko 2023-07-20T02:21:00.784 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/pcmcia/yenta_socket.ko 2023-07-20T02:21:00.784 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/pinctrl 2023-07-20T02:21:00.785 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/pinctrl/intel 2023-07-20T02:21:00.785 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-07-20T02:21:00.785 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/platform 2023-07-20T02:21:00.785 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/platform/x86 2023-07-20T02:21:00.785 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/platform/x86/acer-wmi.ko 2023-07-20T02:21:00.785 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/platform/x86/acerhdf.ko 2023-07-20T02:21:00.785 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/platform/x86/apple-gmux.ko 2023-07-20T02:21:00.785 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/platform/x86/asus-laptop.ko 2023-07-20T02:21:00.786 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-07-20T02:21:00.786 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/platform/x86/asus-wmi.ko 2023-07-20T02:21:00.786 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/platform/x86/classmate-laptop.ko 2023-07-20T02:21:00.786 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-07-20T02:21:00.786 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-07-20T02:21:00.786 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-07-20T02:21:00.786 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-07-20T02:21:00.786 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/platform/x86/hdaps.ko 2023-07-20T02:21:00.787 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/platform/x86/intel 2023-07-20T02:21:00.787 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-07-20T02:21:00.787 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/platform/x86/intel_ips.ko 2023-07-20T02:21:00.787 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/platform/x86/msi-wmi.ko 2023-07-20T02:21:00.787 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/platform/x86/mxm-wmi.ko 2023-07-20T02:21:00.787 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-07-20T02:21:00.787 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/platform/x86/samsung-laptop.ko 2023-07-20T02:21:00.787 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/platform/x86/samsung-q10.ko 2023-07-20T02:21:00.788 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/platform/x86/topstar-laptop.ko 2023-07-20T02:21:00.788 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-07-20T02:21:00.788 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/platform/x86/toshiba_haps.ko 2023-07-20T02:21:00.788 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/platform/x86/wmi-bmof.ko 2023-07-20T02:21:00.788 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/platform/x86/wmi.ko 2023-07-20T02:21:00.788 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/powercap 2023-07-20T02:21:00.788 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/powercap/intel_rapl_common.ko 2023-07-20T02:21:00.788 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/powercap/intel_rapl_msr.ko 2023-07-20T02:21:00.789 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/pps 2023-07-20T02:21:00.789 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/pps/clients 2023-07-20T02:21:00.789 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/pps/clients/pps-gpio.ko 2023-07-20T02:21:00.789 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/pps/clients/pps-ldisc.ko 2023-07-20T02:21:00.789 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/pps/clients/pps_parport.ko 2023-07-20T02:21:00.789 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/pps/pps_core.ko 2023-07-20T02:21:00.789 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ptp 2023-07-20T02:21:00.790 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ptp/ptp.ko 2023-07-20T02:21:00.790 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ptp/ptp_kvm.ko 2023-07-20T02:21:00.790 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/rtc 2023-07-20T02:21:00.790 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/rtc/rtc-bq32k.ko 2023-07-20T02:21:00.790 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/rtc/rtc-bq4802.ko 2023-07-20T02:21:00.790 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/rtc/rtc-ds1286.ko 2023-07-20T02:21:00.790 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/rtc/rtc-ds1307.ko 2023-07-20T02:21:00.790 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/rtc/rtc-ds1374.ko 2023-07-20T02:21:00.791 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/rtc/rtc-ds1511.ko 2023-07-20T02:21:00.791 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/rtc/rtc-ds1553.ko 2023-07-20T02:21:00.791 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/rtc/rtc-ds1672.ko 2023-07-20T02:21:00.791 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/rtc/rtc-ds1742.ko 2023-07-20T02:21:00.791 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/rtc/rtc-ds2404.ko 2023-07-20T02:21:00.791 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/rtc/rtc-ds3232.ko 2023-07-20T02:21:00.791 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/rtc/rtc-em3027.ko 2023-07-20T02:21:00.791 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/rtc/rtc-fm3130.ko 2023-07-20T02:21:00.792 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/rtc/rtc-isl12022.ko 2023-07-20T02:21:00.792 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/rtc/rtc-isl1208.ko 2023-07-20T02:21:00.792 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/rtc/rtc-m41t80.ko 2023-07-20T02:21:00.792 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/rtc/rtc-m48t35.ko 2023-07-20T02:21:00.792 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/rtc/rtc-m48t59.ko 2023-07-20T02:21:00.792 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/rtc/rtc-max6900.ko 2023-07-20T02:21:00.792 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/rtc/rtc-msm6242.ko 2023-07-20T02:21:00.792 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/rtc/rtc-pcf2127.ko 2023-07-20T02:21:00.793 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/rtc/rtc-pcf85063.ko 2023-07-20T02:21:00.793 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/rtc/rtc-pcf8523.ko 2023-07-20T02:21:00.793 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/rtc/rtc-pcf8563.ko 2023-07-20T02:21:00.793 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/rtc/rtc-pcf8583.ko 2023-07-20T02:21:00.793 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/rtc/rtc-rp5c01.ko 2023-07-20T02:21:00.793 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/rtc/rtc-rs5c372.ko 2023-07-20T02:21:00.793 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-07-20T02:21:00.793 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/rtc/rtc-rx8025.ko 2023-07-20T02:21:00.794 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/rtc/rtc-rx8581.ko 2023-07-20T02:21:00.794 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-07-20T02:21:00.794 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/rtc/rtc-x1205.ko 2023-07-20T02:21:00.794 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi 2023-07-20T02:21:00.794 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/3w-9xxx.ko 2023-07-20T02:21:00.794 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/3w-sas.ko 2023-07-20T02:21:00.794 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/3w-xxxx.ko 2023-07-20T02:21:00.794 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/BusLogic.ko 2023-07-20T02:21:00.795 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/a100u2w.ko 2023-07-20T02:21:00.795 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/aacraid 2023-07-20T02:21:00.795 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/aacraid/aacraid.ko 2023-07-20T02:21:00.795 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/advansys.ko 2023-07-20T02:21:00.795 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/aic7xxx 2023-07-20T02:21:00.795 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-07-20T02:21:00.795 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-07-20T02:21:00.795 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/am53c974.ko 2023-07-20T02:21:00.796 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/arcmsr 2023-07-20T02:21:00.796 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-07-20T02:21:00.796 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/atp870u.ko 2023-07-20T02:21:00.796 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/be2iscsi 2023-07-20T02:21:00.796 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-07-20T02:21:00.796 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/bfa 2023-07-20T02:21:00.796 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/bfa/bfa.ko 2023-07-20T02:21:00.797 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/bnx2fc 2023-07-20T02:21:00.797 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-07-20T02:21:00.797 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/bnx2i 2023-07-20T02:21:00.797 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-07-20T02:21:00.797 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/ch.ko 2023-07-20T02:21:00.797 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/csiostor 2023-07-20T02:21:00.797 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/csiostor/csiostor.ko 2023-07-20T02:21:00.797 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/cxgbi 2023-07-20T02:21:00.798 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/cxgbi/cxgb3i 2023-07-20T02:21:00.798 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-07-20T02:21:00.798 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/cxgbi/cxgb4i 2023-07-20T02:21:00.798 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-07-20T02:21:00.798 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-07-20T02:21:00.798 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/dc395x.ko 2023-07-20T02:21:00.798 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/device_handler 2023-07-20T02:21:00.798 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2023-07-20T02:21:00.799 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2023-07-20T02:21:00.799 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2023-07-20T02:21:00.799 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2023-07-20T02:21:00.799 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/dmx3191d.ko 2023-07-20T02:21:00.799 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/esas2r 2023-07-20T02:21:00.799 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/esas2r/esas2r.ko 2023-07-20T02:21:00.799 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/esp_scsi.ko 2023-07-20T02:21:00.799 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/fcoe 2023-07-20T02:21:00.800 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/fcoe/fcoe.ko 2023-07-20T02:21:00.800 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-07-20T02:21:00.800 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/fnic 2023-07-20T02:21:00.800 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/fnic/fnic.ko 2023-07-20T02:21:00.800 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/hpsa.ko 2023-07-20T02:21:00.800 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/hptiop.ko 2023-07-20T02:21:00.800 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/hv_storvsc.ko 2023-07-20T02:21:00.801 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/initio.ko 2023-07-20T02:21:00.801 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/ipr.ko 2023-07-20T02:21:00.801 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/ips.ko 2023-07-20T02:21:00.801 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/isci 2023-07-20T02:21:00.801 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/isci/isci.ko 2023-07-20T02:21:00.801 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-07-20T02:21:00.801 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/iscsi_tcp.ko 2023-07-20T02:21:00.801 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/libfc 2023-07-20T02:21:00.802 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/libfc/libfc.ko 2023-07-20T02:21:00.802 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/libiscsi.ko 2023-07-20T02:21:00.802 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/libiscsi_tcp.ko 2023-07-20T02:21:00.802 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/libsas 2023-07-20T02:21:00.802 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/libsas/libsas.ko 2023-07-20T02:21:00.802 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/lpfc 2023-07-20T02:21:00.802 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/lpfc/lpfc.ko 2023-07-20T02:21:00.802 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/megaraid 2023-07-20T02:21:00.803 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/megaraid.ko 2023-07-20T02:21:00.803 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-07-20T02:21:00.803 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-07-20T02:21:00.803 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-07-20T02:21:00.803 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/mpt3sas 2023-07-20T02:21:00.803 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-07-20T02:21:00.803 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/mvsas 2023-07-20T02:21:00.803 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/mvsas/mvsas.ko 2023-07-20T02:21:00.804 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/mvumi.ko 2023-07-20T02:21:00.804 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/pm8001 2023-07-20T02:21:00.804 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-07-20T02:21:00.804 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/pmcraid.ko 2023-07-20T02:21:00.804 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/qla1280.ko 2023-07-20T02:21:00.804 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/qla2xxx 2023-07-20T02:21:00.804 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-07-20T02:21:00.804 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-07-20T02:21:00.805 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/qla4xxx 2023-07-20T02:21:00.805 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-07-20T02:21:00.805 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/raid_class.ko 2023-07-20T02:21:00.805 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/scsi_debug.ko 2023-07-20T02:21:00.805 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/scsi_transport_fc.ko 2023-07-20T02:21:00.805 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-07-20T02:21:00.805 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/scsi_transport_sas.ko 2023-07-20T02:21:00.806 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/scsi_transport_spi.ko 2023-07-20T02:21:00.806 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/scsi_transport_srp.ko 2023-07-20T02:21:00.806 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/ses.ko 2023-07-20T02:21:00.806 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/st.ko 2023-07-20T02:21:00.806 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/stex.ko 2023-07-20T02:21:00.806 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/sym53c8xx_2 2023-07-20T02:21:00.806 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-07-20T02:21:00.806 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/virtio_scsi.ko 2023-07-20T02:21:00.807 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/vmw_pvscsi.ko 2023-07-20T02:21:00.807 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/scsi/wd719x.ko 2023-07-20T02:21:00.807 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ssb 2023-07-20T02:21:00.807 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ssb/ssb.ko 2023-07-20T02:21:00.807 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/target 2023-07-20T02:21:00.807 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/target/iscsi 2023-07-20T02:21:00.807 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-07-20T02:21:00.807 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/target/loopback 2023-07-20T02:21:00.808 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/target/loopback/tcm_loop.ko 2023-07-20T02:21:00.808 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/target/target_core_file.ko 2023-07-20T02:21:00.808 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/target/target_core_iblock.ko 2023-07-20T02:21:00.808 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/target/target_core_mod.ko 2023-07-20T02:21:00.808 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/target/target_core_pscsi.ko 2023-07-20T02:21:00.808 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/target/target_core_user.ko 2023-07-20T02:21:00.808 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/target/tcm_fc 2023-07-20T02:21:00.808 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-07-20T02:21:00.809 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/thermal 2023-07-20T02:21:00.809 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/thermal/intel 2023-07-20T02:21:00.809 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/thermal/intel/int340x_thermal 2023-07-20T02:21:00.809 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-07-20T02:21:00.809 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-07-20T02:21:00.809 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-07-20T02:21:00.809 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-07-20T02:21:00.809 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-07-20T02:21:00.810 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-07-20T02:21:00.810 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-07-20T02:21:00.810 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-07-20T02:21:00.810 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-07-20T02:21:00.810 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-07-20T02:21:00.810 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-07-20T02:21:00.810 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-07-20T02:21:00.811 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-07-20T02:21:00.811 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-07-20T02:21:00.811 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-07-20T02:21:00.811 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/tty 2023-07-20T02:21:00.811 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/tty/ipwireless 2023-07-20T02:21:00.811 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-07-20T02:21:00.811 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/tty/n_gsm.ko 2023-07-20T02:21:00.811 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/tty/n_hdlc.ko 2023-07-20T02:21:00.812 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/tty/nozomi.ko 2023-07-20T02:21:00.812 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/tty/serial 2023-07-20T02:21:00.812 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/tty/serial/8250 2023-07-20T02:21:00.812 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-07-20T02:21:00.812 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/tty/serial/arc_uart.ko 2023-07-20T02:21:00.812 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/tty/serial/jsm 2023-07-20T02:21:00.812 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/tty/serial/jsm/jsm.ko 2023-07-20T02:21:00.812 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/tty/synclink_gt.ko 2023-07-20T02:21:00.813 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ufs 2023-07-20T02:21:00.813 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ufs/core 2023-07-20T02:21:00.813 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ufs/core/ufshcd-core.ko 2023-07-20T02:21:00.813 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ufs/host 2023-07-20T02:21:00.813 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-07-20T02:21:00.813 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/uio 2023-07-20T02:21:00.813 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/uio/uio.ko 2023-07-20T02:21:00.813 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/uio/uio_aec.ko 2023-07-20T02:21:00.814 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/uio/uio_cif.ko 2023-07-20T02:21:00.814 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/uio/uio_pci_generic.ko 2023-07-20T02:21:00.814 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/uio/uio_sercos3.ko 2023-07-20T02:21:00.814 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb 2023-07-20T02:21:00.814 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/class 2023-07-20T02:21:00.814 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/class/cdc-acm.ko 2023-07-20T02:21:00.814 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/class/cdc-wdm.ko 2023-07-20T02:21:00.814 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/class/usblp.ko 2023-07-20T02:21:00.815 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/class/usbtmc.ko 2023-07-20T02:21:00.815 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/host 2023-07-20T02:21:00.815 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/host/sl811-hcd.ko 2023-07-20T02:21:00.815 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/image 2023-07-20T02:21:00.815 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/image/mdc800.ko 2023-07-20T02:21:00.815 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/image/microtek.ko 2023-07-20T02:21:00.815 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/misc 2023-07-20T02:21:00.815 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/misc/adutux.ko 2023-07-20T02:21:00.816 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/misc/appledisplay.ko 2023-07-20T02:21:00.816 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/misc/emi26.ko 2023-07-20T02:21:00.816 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/misc/emi62.ko 2023-07-20T02:21:00.816 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/misc/ezusb.ko 2023-07-20T02:21:00.816 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/misc/idmouse.ko 2023-07-20T02:21:00.816 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/misc/iowarrior.ko 2023-07-20T02:21:00.816 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/misc/isight_firmware.ko 2023-07-20T02:21:00.817 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/misc/ldusb.ko 2023-07-20T02:21:00.817 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/misc/legousbtower.ko 2023-07-20T02:21:00.817 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/misc/sisusbvga 2023-07-20T02:21:00.817 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-07-20T02:21:00.817 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/misc/trancevibrator.ko 2023-07-20T02:21:00.817 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/misc/usb3503.ko 2023-07-20T02:21:00.817 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/misc/usblcd.ko 2023-07-20T02:21:00.817 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/misc/usbsevseg.ko 2023-07-20T02:21:00.818 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/misc/uss720.ko 2023-07-20T02:21:00.818 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/misc/yurex.ko 2023-07-20T02:21:00.818 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/phy 2023-07-20T02:21:00.818 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/phy/phy-generic.ko 2023-07-20T02:21:00.818 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial 2023-07-20T02:21:00.818 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/aircable.ko 2023-07-20T02:21:00.818 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/ark3116.ko 2023-07-20T02:21:00.818 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/belkin_sa.ko 2023-07-20T02:21:00.819 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/ch341.ko 2023-07-20T02:21:00.819 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/cp210x.ko 2023-07-20T02:21:00.819 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/cyberjack.ko 2023-07-20T02:21:00.819 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/cypress_m8.ko 2023-07-20T02:21:00.819 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/digi_acceleport.ko 2023-07-20T02:21:00.819 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/empeg.ko 2023-07-20T02:21:00.819 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/ftdi_sio.ko 2023-07-20T02:21:00.819 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/garmin_gps.ko 2023-07-20T02:21:00.820 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/io_edgeport.ko 2023-07-20T02:21:00.820 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/io_ti.ko 2023-07-20T02:21:00.820 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/ipaq.ko 2023-07-20T02:21:00.820 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/ipw.ko 2023-07-20T02:21:00.820 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/ir-usb.ko 2023-07-20T02:21:00.820 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-07-20T02:21:00.820 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/keyspan.ko 2023-07-20T02:21:00.821 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/keyspan_pda.ko 2023-07-20T02:21:00.821 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/kl5kusb105.ko 2023-07-20T02:21:00.821 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/kobil_sct.ko 2023-07-20T02:21:00.821 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/mct_u232.ko 2023-07-20T02:21:00.821 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/mos7720.ko 2023-07-20T02:21:00.821 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/mos7840.ko 2023-07-20T02:21:00.821 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/navman.ko 2023-07-20T02:21:00.821 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/omninet.ko 2023-07-20T02:21:00.822 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/opticon.ko 2023-07-20T02:21:00.822 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/option.ko 2023-07-20T02:21:00.822 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/oti6858.ko 2023-07-20T02:21:00.822 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/pl2303.ko 2023-07-20T02:21:00.822 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/qcaux.ko 2023-07-20T02:21:00.822 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/qcserial.ko 2023-07-20T02:21:00.822 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/quatech2.ko 2023-07-20T02:21:00.822 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/safe_serial.ko 2023-07-20T02:21:00.823 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/sierra.ko 2023-07-20T02:21:00.823 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/spcp8x5.ko 2023-07-20T02:21:00.823 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/ssu100.ko 2023-07-20T02:21:00.823 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/symbolserial.ko 2023-07-20T02:21:00.823 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-07-20T02:21:00.823 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-07-20T02:21:00.823 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/usb_debug.ko 2023-07-20T02:21:00.823 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/usb_wwan.ko 2023-07-20T02:21:00.824 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/visor.ko 2023-07-20T02:21:00.824 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/whiteheat.ko 2023-07-20T02:21:00.824 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/serial/xsens_mt.ko 2023-07-20T02:21:00.824 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/storage 2023-07-20T02:21:00.824 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/storage/uas.ko 2023-07-20T02:21:00.824 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/storage/ums-alauda.ko 2023-07-20T02:21:00.824 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/storage/ums-cypress.ko 2023-07-20T02:21:00.824 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/storage/ums-datafab.ko 2023-07-20T02:21:00.825 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-07-20T02:21:00.825 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/storage/ums-freecom.ko 2023-07-20T02:21:00.825 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/storage/ums-isd200.ko 2023-07-20T02:21:00.825 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-07-20T02:21:00.825 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/storage/ums-karma.ko 2023-07-20T02:21:00.825 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/storage/ums-onetouch.ko 2023-07-20T02:21:00.825 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/storage/ums-realtek.ko 2023-07-20T02:21:00.826 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/storage/ums-sddr09.ko 2023-07-20T02:21:00.826 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/storage/ums-sddr55.ko 2023-07-20T02:21:00.826 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/storage/ums-usbat.ko 2023-07-20T02:21:00.826 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/storage/usb-storage.ko 2023-07-20T02:21:00.826 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/usbip 2023-07-20T02:21:00.826 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/usbip/usbip-core.ko 2023-07-20T02:21:00.826 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/usbip/usbip-host.ko 2023-07-20T02:21:00.826 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-07-20T02:21:00.827 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/vfio 2023-07-20T02:21:00.827 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/vfio/pci 2023-07-20T02:21:00.827 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-07-20T02:21:00.827 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/vfio/pci/vfio-pci.ko 2023-07-20T02:21:00.827 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/vfio/vfio.ko 2023-07-20T02:21:00.827 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-07-20T02:21:00.827 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/vhost 2023-07-20T02:21:00.827 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/vhost/vhost.ko 2023-07-20T02:21:00.828 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/vhost/vhost_iotlb.ko 2023-07-20T02:21:00.828 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/vhost/vhost_net.ko 2023-07-20T02:21:00.828 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/vhost/vhost_scsi.ko 2023-07-20T02:21:00.828 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/video 2023-07-20T02:21:00.828 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/video/backlight 2023-07-20T02:21:00.828 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/video/backlight/apple_bl.ko 2023-07-20T02:21:00.828 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/video/backlight/lcd.ko 2023-07-20T02:21:00.828 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/video/backlight/platform_lcd.ko 2023-07-20T02:21:00.829 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/virtio 2023-07-20T02:21:00.829 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/virtio/virtio.ko 2023-07-20T02:21:00.829 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/virtio/virtio_balloon.ko 2023-07-20T02:21:00.829 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/virtio/virtio_mmio.ko 2023-07-20T02:21:00.829 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/virtio/virtio_pci.ko 2023-07-20T02:21:00.829 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2023-07-20T02:21:00.829 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2023-07-20T02:21:00.829 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/virtio/virtio_ring.ko 2023-07-20T02:21:00.830 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/w1 2023-07-20T02:21:00.830 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/w1/masters 2023-07-20T02:21:00.830 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/w1/masters/ds2482.ko 2023-07-20T02:21:00.830 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/w1/masters/ds2490.ko 2023-07-20T02:21:00.830 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/w1/slaves 2023-07-20T02:21:00.830 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-07-20T02:21:00.830 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-07-20T02:21:00.830 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-07-20T02:21:00.831 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-07-20T02:21:00.831 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-07-20T02:21:00.831 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-07-20T02:21:00.831 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-07-20T02:21:00.831 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-07-20T02:21:00.831 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-07-20T02:21:00.831 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/w1/slaves/w1_smem.ko 2023-07-20T02:21:00.831 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/w1/slaves/w1_therm.ko 2023-07-20T02:21:00.832 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/w1/wire.ko 2023-07-20T02:21:00.832 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/watchdog 2023-07-20T02:21:00.832 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/watchdog/alim1535_wdt.ko 2023-07-20T02:21:00.832 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/watchdog/alim7101_wdt.ko 2023-07-20T02:21:00.832 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/watchdog/f71808e_wdt.ko 2023-07-20T02:21:00.832 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/watchdog/hpwdt.ko 2023-07-20T02:21:00.832 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/watchdog/i6300esb.ko 2023-07-20T02:21:00.833 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-07-20T02:21:00.833 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/watchdog/iTCO_wdt.ko 2023-07-20T02:21:00.833 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/watchdog/ib700wdt.ko 2023-07-20T02:21:00.833 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/watchdog/ibmasr.ko 2023-07-20T02:21:00.833 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-07-20T02:21:00.833 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/watchdog/it8712f_wdt.ko 2023-07-20T02:21:00.833 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/watchdog/it87_wdt.ko 2023-07-20T02:21:00.833 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/watchdog/machzwd.ko 2023-07-20T02:21:00.834 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/watchdog/nv_tco.ko 2023-07-20T02:21:00.834 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/watchdog/pcwd_pci.ko 2023-07-20T02:21:00.834 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/watchdog/pcwd_usb.ko 2023-07-20T02:21:00.834 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-07-20T02:21:00.834 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/watchdog/sch311x_wdt.ko 2023-07-20T02:21:00.834 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/watchdog/softdog.ko 2023-07-20T02:21:00.834 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/watchdog/sp5100_tco.ko 2023-07-20T02:21:00.834 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/watchdog/via_wdt.ko 2023-07-20T02:21:00.835 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-07-20T02:21:00.835 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/watchdog/w83877f_wdt.ko 2023-07-20T02:21:00.835 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/watchdog/w83977f_wdt.ko 2023-07-20T02:21:00.835 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/watchdog/wdt_pci.ko 2023-07-20T02:21:00.835 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/watchdog/xen_wdt.ko 2023-07-20T02:21:00.835 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/xen 2023-07-20T02:21:00.835 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/xen/xen-acpi-processor.ko 2023-07-20T02:21:00.835 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/xen/xen-evtchn.ko 2023-07-20T02:21:00.836 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/xen/xen-gntalloc.ko 2023-07-20T02:21:00.836 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/xen/xen-gntdev.ko 2023-07-20T02:21:00.836 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/xen/xen-pciback 2023-07-20T02:21:00.836 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-07-20T02:21:00.836 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/xen/xen-privcmd.ko 2023-07-20T02:21:00.836 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/xen/xenfs 2023-07-20T02:21:00.836 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/drivers/xen/xenfs/xenfs.ko 2023-07-20T02:21:00.836 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs 2023-07-20T02:21:00.837 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/9p 2023-07-20T02:21:00.837 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/9p/9p.ko 2023-07-20T02:21:00.837 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/affs 2023-07-20T02:21:00.837 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/affs/affs.ko 2023-07-20T02:21:00.837 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/befs 2023-07-20T02:21:00.837 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/befs/befs.ko 2023-07-20T02:21:00.837 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/binfmt_misc.ko 2023-07-20T02:21:00.837 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/btrfs 2023-07-20T02:21:00.838 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/btrfs/btrfs.ko 2023-07-20T02:21:00.838 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/cachefiles 2023-07-20T02:21:00.838 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/cachefiles/cachefiles.ko 2023-07-20T02:21:00.838 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/ceph 2023-07-20T02:21:00.838 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/ceph/ceph.ko 2023-07-20T02:21:00.838 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/coda 2023-07-20T02:21:00.838 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/coda/coda.ko 2023-07-20T02:21:00.838 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/cramfs 2023-07-20T02:21:00.839 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/cramfs/cramfs.ko 2023-07-20T02:21:00.839 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/dlm 2023-07-20T02:21:00.839 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/dlm/dlm.ko 2023-07-20T02:21:00.839 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/ecryptfs 2023-07-20T02:21:00.839 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/ecryptfs/ecryptfs.ko 2023-07-20T02:21:00.839 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/f2fs 2023-07-20T02:21:00.839 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/f2fs/f2fs.ko 2023-07-20T02:21:00.839 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/fat 2023-07-20T02:21:00.840 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/fat/fat.ko 2023-07-20T02:21:00.840 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/fat/msdos.ko 2023-07-20T02:21:00.840 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/fat/vfat.ko 2023-07-20T02:21:00.840 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/fscache 2023-07-20T02:21:00.840 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/fscache/fscache.ko 2023-07-20T02:21:00.840 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/fuse 2023-07-20T02:21:00.840 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/fuse/cuse.ko 2023-07-20T02:21:00.841 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/fuse/fuse.ko 2023-07-20T02:21:00.841 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/gfs2 2023-07-20T02:21:00.841 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/gfs2/gfs2.ko 2023-07-20T02:21:00.841 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/hfs 2023-07-20T02:21:00.841 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/hfs/hfs.ko 2023-07-20T02:21:00.841 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/hfsplus 2023-07-20T02:21:00.841 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/hfsplus/hfsplus.ko 2023-07-20T02:21:00.841 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/isofs 2023-07-20T02:21:00.842 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/isofs/isofs.ko 2023-07-20T02:21:00.842 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/jfs 2023-07-20T02:21:00.842 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/jfs/jfs.ko 2023-07-20T02:21:00.842 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/lockd 2023-07-20T02:21:00.842 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/lockd/lockd.ko 2023-07-20T02:21:00.842 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/minix 2023-07-20T02:21:00.842 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/minix/minix.ko 2023-07-20T02:21:00.842 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/netfs 2023-07-20T02:21:00.843 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/netfs/netfs.ko 2023-07-20T02:21:00.843 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nfs 2023-07-20T02:21:00.843 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nfs/blocklayout 2023-07-20T02:21:00.843 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-07-20T02:21:00.843 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nfs/filelayout 2023-07-20T02:21:00.843 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-07-20T02:21:00.843 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nfs/flexfilelayout 2023-07-20T02:21:00.843 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-07-20T02:21:00.844 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nfs/nfs.ko 2023-07-20T02:21:00.844 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nfs/nfsv3.ko 2023-07-20T02:21:00.844 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nfs/nfsv4.ko 2023-07-20T02:21:00.844 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nfs_common 2023-07-20T02:21:00.844 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nfs_common/grace.ko 2023-07-20T02:21:00.844 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nfs_common/nfs_acl.ko 2023-07-20T02:21:00.844 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nfsd 2023-07-20T02:21:00.844 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nfsd/nfsd.ko 2023-07-20T02:21:00.845 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nilfs2 2023-07-20T02:21:00.845 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nilfs2/nilfs2.ko 2023-07-20T02:21:00.845 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls 2023-07-20T02:21:00.845 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/mac-celtic.ko 2023-07-20T02:21:00.845 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/mac-centeuro.ko 2023-07-20T02:21:00.845 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/mac-croatian.ko 2023-07-20T02:21:00.845 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/mac-cyrillic.ko 2023-07-20T02:21:00.845 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/mac-gaelic.ko 2023-07-20T02:21:00.846 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/mac-greek.ko 2023-07-20T02:21:00.846 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/mac-iceland.ko 2023-07-20T02:21:00.846 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/mac-inuit.ko 2023-07-20T02:21:00.846 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/mac-roman.ko 2023-07-20T02:21:00.846 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/mac-romanian.ko 2023-07-20T02:21:00.846 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/mac-turkish.ko 2023-07-20T02:21:00.846 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_cp1250.ko 2023-07-20T02:21:00.846 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_cp1251.ko 2023-07-20T02:21:00.847 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_cp1255.ko 2023-07-20T02:21:00.847 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_cp737.ko 2023-07-20T02:21:00.847 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_cp775.ko 2023-07-20T02:21:00.847 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_cp850.ko 2023-07-20T02:21:00.847 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_cp852.ko 2023-07-20T02:21:00.847 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_cp855.ko 2023-07-20T02:21:00.847 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_cp857.ko 2023-07-20T02:21:00.847 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_cp860.ko 2023-07-20T02:21:00.848 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_cp861.ko 2023-07-20T02:21:00.848 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_cp862.ko 2023-07-20T02:21:00.848 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_cp863.ko 2023-07-20T02:21:00.848 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_cp864.ko 2023-07-20T02:21:00.848 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_cp865.ko 2023-07-20T02:21:00.848 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_cp866.ko 2023-07-20T02:21:00.848 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_cp869.ko 2023-07-20T02:21:00.849 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_cp874.ko 2023-07-20T02:21:00.849 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_cp932.ko 2023-07-20T02:21:00.849 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_cp936.ko 2023-07-20T02:21:00.849 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_cp949.ko 2023-07-20T02:21:00.849 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_cp950.ko 2023-07-20T02:21:00.849 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_euc-jp.ko 2023-07-20T02:21:00.849 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_iso8859-1.ko 2023-07-20T02:21:00.849 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_iso8859-13.ko 2023-07-20T02:21:00.850 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_iso8859-14.ko 2023-07-20T02:21:00.850 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_iso8859-15.ko 2023-07-20T02:21:00.850 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_iso8859-2.ko 2023-07-20T02:21:00.850 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_iso8859-3.ko 2023-07-20T02:21:00.850 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_iso8859-4.ko 2023-07-20T02:21:00.850 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_iso8859-5.ko 2023-07-20T02:21:00.850 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_iso8859-6.ko 2023-07-20T02:21:00.850 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_iso8859-7.ko 2023-07-20T02:21:00.851 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_iso8859-9.ko 2023-07-20T02:21:00.851 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_koi8-r.ko 2023-07-20T02:21:00.851 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_koi8-ru.ko 2023-07-20T02:21:00.851 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_koi8-u.ko 2023-07-20T02:21:00.851 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/nls/nls_utf8.ko 2023-07-20T02:21:00.851 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/ocfs2 2023-07-20T02:21:00.851 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/ocfs2/cluster 2023-07-20T02:21:00.851 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-07-20T02:21:00.852 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/ocfs2/dlm 2023-07-20T02:21:00.852 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-07-20T02:21:00.852 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/ocfs2/dlmfs 2023-07-20T02:21:00.852 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-07-20T02:21:00.852 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/ocfs2/ocfs2.ko 2023-07-20T02:21:00.852 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-07-20T02:21:00.852 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-07-20T02:21:00.853 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-07-20T02:21:00.853 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/overlayfs 2023-07-20T02:21:00.853 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/overlayfs/overlay.ko 2023-07-20T02:21:00.853 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/pstore 2023-07-20T02:21:00.853 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/pstore/ramoops.ko 2023-07-20T02:21:00.853 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/reiserfs 2023-07-20T02:21:00.853 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/reiserfs/reiserfs.ko 2023-07-20T02:21:00.853 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/romfs 2023-07-20T02:21:00.854 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/romfs/romfs.ko 2023-07-20T02:21:00.854 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/smb 2023-07-20T02:21:00.854 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/smb/client 2023-07-20T02:21:00.854 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/smb/client/cifs.ko 2023-07-20T02:21:00.854 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/smb/common 2023-07-20T02:21:00.854 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/smb/common/cifs_arc4.ko 2023-07-20T02:21:00.854 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/smb/common/cifs_md4.ko 2023-07-20T02:21:00.854 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/squashfs 2023-07-20T02:21:00.855 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/squashfs/squashfs.ko 2023-07-20T02:21:00.855 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/sysv 2023-07-20T02:21:00.855 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/sysv/sysv.ko 2023-07-20T02:21:00.855 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/udf 2023-07-20T02:21:00.855 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/udf/udf.ko 2023-07-20T02:21:00.855 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/ufs 2023-07-20T02:21:00.855 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/ufs/ufs.ko 2023-07-20T02:21:00.855 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/xfs 2023-07-20T02:21:00.856 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/fs/xfs/xfs.ko 2023-07-20T02:21:00.856 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/kernel 2023-07-20T02:21:00.856 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/kernel/rcu 2023-07-20T02:21:00.856 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/kernel/rcu/rcutorture.ko 2023-07-20T02:21:00.856 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/kernel/torture.ko 2023-07-20T02:21:00.856 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/kernel/trace 2023-07-20T02:21:00.856 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/kernel/trace/ring_buffer_benchmark.ko 2023-07-20T02:21:00.856 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/lib 2023-07-20T02:21:00.857 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/lib/asn1_encoder.ko 2023-07-20T02:21:00.857 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/lib/crc-itu-t.ko 2023-07-20T02:21:00.857 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/lib/crc8.ko 2023-07-20T02:21:00.857 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/lib/crypto 2023-07-20T02:21:00.857 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/lib/crypto/libarc4.ko 2023-07-20T02:21:00.857 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/lib/crypto/libdes.ko 2023-07-20T02:21:00.857 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/lib/libcrc32c.ko 2023-07-20T02:21:00.858 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/lib/lru_cache.ko 2023-07-20T02:21:00.858 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/lib/lz4 2023-07-20T02:21:00.858 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/lib/lz4/lz4_compress.ko 2023-07-20T02:21:00.858 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/lib/lz4/lz4hc_compress.ko 2023-07-20T02:21:00.858 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/lib/math 2023-07-20T02:21:00.858 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/lib/math/cordic.ko 2023-07-20T02:21:00.858 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/lib/raid6 2023-07-20T02:21:00.858 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/lib/raid6/raid6_pq.ko 2023-07-20T02:21:00.859 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/lib/reed_solomon 2023-07-20T02:21:00.859 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/lib/reed_solomon/reed_solomon.ko 2023-07-20T02:21:00.859 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/lib/ts_bm.ko 2023-07-20T02:21:00.859 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/lib/ts_fsm.ko 2023-07-20T02:21:00.859 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/lib/ts_kmp.ko 2023-07-20T02:21:00.859 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/lib/zstd 2023-07-20T02:21:00.859 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/lib/zstd/zstd_compress.ko 2023-07-20T02:21:00.859 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/mm 2023-07-20T02:21:00.860 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/mm/hwpoison-inject.ko 2023-07-20T02:21:00.860 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net 2023-07-20T02:21:00.860 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/6lowpan 2023-07-20T02:21:00.860 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/6lowpan/6lowpan.ko 2023-07-20T02:21:00.860 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/6lowpan/nhc_dest.ko 2023-07-20T02:21:00.860 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/6lowpan/nhc_fragment.ko 2023-07-20T02:21:00.860 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/6lowpan/nhc_hop.ko 2023-07-20T02:21:00.861 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/6lowpan/nhc_ipv6.ko 2023-07-20T02:21:00.861 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/6lowpan/nhc_mobility.ko 2023-07-20T02:21:00.861 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/6lowpan/nhc_routing.ko 2023-07-20T02:21:00.861 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/6lowpan/nhc_udp.ko 2023-07-20T02:21:00.861 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/802 2023-07-20T02:21:00.861 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/802/garp.ko 2023-07-20T02:21:00.861 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/802/mrp.ko 2023-07-20T02:21:00.861 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/802/p8022.ko 2023-07-20T02:21:00.862 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/802/psnap.ko 2023-07-20T02:21:00.862 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/802/stp.ko 2023-07-20T02:21:00.862 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/8021q 2023-07-20T02:21:00.862 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/8021q/8021q.ko 2023-07-20T02:21:00.862 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/9p 2023-07-20T02:21:00.862 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/9p/9pnet.ko 2023-07-20T02:21:00.862 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/9p/9pnet_fd.ko 2023-07-20T02:21:00.862 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/9p/9pnet_rdma.ko 2023-07-20T02:21:00.863 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/9p/9pnet_virtio.ko 2023-07-20T02:21:00.863 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/batman-adv 2023-07-20T02:21:00.863 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/batman-adv/batman-adv.ko 2023-07-20T02:21:00.863 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/bridge 2023-07-20T02:21:00.863 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/bridge/br_netfilter.ko 2023-07-20T02:21:00.863 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/bridge/bridge.ko 2023-07-20T02:21:00.863 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/bridge/netfilter 2023-07-20T02:21:00.863 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-07-20T02:21:00.864 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/bridge/netfilter/ebt_among.ko 2023-07-20T02:21:00.864 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/bridge/netfilter/ebt_arp.ko 2023-07-20T02:21:00.864 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-07-20T02:21:00.864 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-07-20T02:21:00.864 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/bridge/netfilter/ebt_ip.ko 2023-07-20T02:21:00.864 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-07-20T02:21:00.864 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/bridge/netfilter/ebt_limit.ko 2023-07-20T02:21:00.864 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/bridge/netfilter/ebt_log.ko 2023-07-20T02:21:00.865 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/bridge/netfilter/ebt_mark.ko 2023-07-20T02:21:00.865 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-07-20T02:21:00.865 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-07-20T02:21:00.865 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-07-20T02:21:00.865 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-07-20T02:21:00.865 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/bridge/netfilter/ebt_snat.ko 2023-07-20T02:21:00.865 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/bridge/netfilter/ebt_stp.ko 2023-07-20T02:21:00.865 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-07-20T02:21:00.866 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-07-20T02:21:00.866 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-07-20T02:21:00.866 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-07-20T02:21:00.866 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/bridge/netfilter/ebtables.ko 2023-07-20T02:21:00.866 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2023-07-20T02:21:00.866 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-07-20T02:21:00.866 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-07-20T02:21:00.866 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ceph 2023-07-20T02:21:00.867 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ceph/libceph.ko 2023-07-20T02:21:00.867 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/core 2023-07-20T02:21:00.867 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/core/failover.ko 2023-07-20T02:21:00.867 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/core/pktgen.ko 2023-07-20T02:21:00.867 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/dccp 2023-07-20T02:21:00.867 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/dccp/dccp.ko 2023-07-20T02:21:00.867 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/dccp/dccp_diag.ko 2023-07-20T02:21:00.867 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/dccp/dccp_ipv4.ko 2023-07-20T02:21:00.868 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/dccp/dccp_ipv6.ko 2023-07-20T02:21:00.868 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/dns_resolver 2023-07-20T02:21:00.868 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/dns_resolver/dns_resolver.ko 2023-07-20T02:21:00.868 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/dsa 2023-07-20T02:21:00.868 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/dsa/dsa_core.ko 2023-07-20T02:21:00.868 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/dsa/tag_brcm.ko 2023-07-20T02:21:00.868 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/dsa/tag_dsa.ko 2023-07-20T02:21:00.869 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/dsa/tag_none.ko 2023-07-20T02:21:00.869 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/dsa/tag_trailer.ko 2023-07-20T02:21:00.869 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ieee802154 2023-07-20T02:21:00.869 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ieee802154/6lowpan 2023-07-20T02:21:00.869 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-07-20T02:21:00.869 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ieee802154/ieee802154.ko 2023-07-20T02:21:00.869 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ieee802154/ieee802154_socket.ko 2023-07-20T02:21:00.869 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4 2023-07-20T02:21:00.870 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/ah4.ko 2023-07-20T02:21:00.870 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/esp4.ko 2023-07-20T02:21:00.870 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/fou.ko 2023-07-20T02:21:00.870 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/gre.ko 2023-07-20T02:21:00.870 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/inet_diag.ko 2023-07-20T02:21:00.870 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/ip_gre.ko 2023-07-20T02:21:00.870 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/ip_tunnel.ko 2023-07-20T02:21:00.870 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/ip_vti.ko 2023-07-20T02:21:00.871 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/ipcomp.ko 2023-07-20T02:21:00.871 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/ipip.ko 2023-07-20T02:21:00.871 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/netfilter 2023-07-20T02:21:00.871 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/netfilter/arp_tables.ko 2023-07-20T02:21:00.871 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-07-20T02:21:00.871 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-07-20T02:21:00.871 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-07-20T02:21:00.872 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-07-20T02:21:00.872 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-07-20T02:21:00.872 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-07-20T02:21:00.872 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-07-20T02:21:00.872 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-07-20T02:21:00.872 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-07-20T02:21:00.872 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/netfilter/iptable_security.ko 2023-07-20T02:21:00.872 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-07-20T02:21:00.873 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-07-20T02:21:00.873 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-07-20T02:21:00.873 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-07-20T02:21:00.873 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-07-20T02:21:00.873 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-07-20T02:21:00.873 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-07-20T02:21:00.873 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-07-20T02:21:00.873 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-07-20T02:21:00.874 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-07-20T02:21:00.874 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/tcp_bic.ko 2023-07-20T02:21:00.874 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/tcp_dctcp.ko 2023-07-20T02:21:00.874 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/tcp_diag.ko 2023-07-20T02:21:00.874 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/tcp_highspeed.ko 2023-07-20T02:21:00.874 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/tcp_htcp.ko 2023-07-20T02:21:00.874 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/tcp_hybla.ko 2023-07-20T02:21:00.874 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/tcp_illinois.ko 2023-07-20T02:21:00.875 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/tcp_lp.ko 2023-07-20T02:21:00.875 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/tcp_scalable.ko 2023-07-20T02:21:00.875 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/tcp_vegas.ko 2023-07-20T02:21:00.875 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/tcp_veno.ko 2023-07-20T02:21:00.875 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/tcp_westwood.ko 2023-07-20T02:21:00.875 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/tcp_yeah.ko 2023-07-20T02:21:00.875 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/tunnel4.ko 2023-07-20T02:21:00.875 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/udp_diag.ko 2023-07-20T02:21:00.876 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/udp_tunnel.ko 2023-07-20T02:21:00.876 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv4/xfrm4_tunnel.ko 2023-07-20T02:21:00.876 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6 2023-07-20T02:21:00.876 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/ah6.ko 2023-07-20T02:21:00.876 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/esp6.ko 2023-07-20T02:21:00.876 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/fou6.ko 2023-07-20T02:21:00.876 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/ip6_tunnel.ko 2023-07-20T02:21:00.877 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-07-20T02:21:00.877 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/ip6_vti.ko 2023-07-20T02:21:00.877 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/ipcomp6.ko 2023-07-20T02:21:00.877 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/netfilter 2023-07-20T02:21:00.877 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-07-20T02:21:00.877 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-07-20T02:21:00.877 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-07-20T02:21:00.877 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-07-20T02:21:00.878 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-07-20T02:21:00.878 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-07-20T02:21:00.878 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-07-20T02:21:00.878 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-07-20T02:21:00.878 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-07-20T02:21:00.878 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-07-20T02:21:00.878 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-07-20T02:21:00.878 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-07-20T02:21:00.879 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-07-20T02:21:00.879 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-07-20T02:21:00.879 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-07-20T02:21:00.879 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-07-20T02:21:00.879 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-07-20T02:21:00.879 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-07-20T02:21:00.879 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-07-20T02:21:00.879 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-07-20T02:21:00.880 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-07-20T02:21:00.880 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-07-20T02:21:00.880 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-07-20T02:21:00.880 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-07-20T02:21:00.880 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/sit.ko 2023-07-20T02:21:00.880 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/tunnel6.ko 2023-07-20T02:21:00.880 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/ipv6/xfrm6_tunnel.ko 2023-07-20T02:21:00.880 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/key 2023-07-20T02:21:00.881 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/key/af_key.ko 2023-07-20T02:21:00.881 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/l2tp 2023-07-20T02:21:00.881 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/l2tp/l2tp_core.ko 2023-07-20T02:21:00.881 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/l2tp/l2tp_debugfs.ko 2023-07-20T02:21:00.881 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/l2tp/l2tp_eth.ko 2023-07-20T02:21:00.881 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/l2tp/l2tp_ip.ko 2023-07-20T02:21:00.881 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/l2tp/l2tp_ip6.ko 2023-07-20T02:21:00.881 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/l2tp/l2tp_netlink.ko 2023-07-20T02:21:00.882 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/l2tp/l2tp_ppp.ko 2023-07-20T02:21:00.882 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/llc 2023-07-20T02:21:00.882 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/llc/llc.ko 2023-07-20T02:21:00.882 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/mac802154 2023-07-20T02:21:00.882 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/mac802154/mac802154.ko 2023-07-20T02:21:00.882 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/mpls 2023-07-20T02:21:00.882 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/mpls/mpls_gso.ko 2023-07-20T02:21:00.883 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter 2023-07-20T02:21:00.883 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/ipset 2023-07-20T02:21:00.883 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/ipset/ip_set.ko 2023-07-20T02:21:00.883 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-07-20T02:21:00.883 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-07-20T02:21:00.883 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-07-20T02:21:00.883 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-07-20T02:21:00.883 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-07-20T02:21:00.884 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-07-20T02:21:00.884 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-07-20T02:21:00.884 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-07-20T02:21:00.884 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-07-20T02:21:00.884 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-07-20T02:21:00.884 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-07-20T02:21:00.884 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-07-20T02:21:00.884 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-07-20T02:21:00.885 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-07-20T02:21:00.885 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-07-20T02:21:00.885 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/ipvs 2023-07-20T02:21:00.885 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/ipvs/ip_vs.ko 2023-07-20T02:21:00.885 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-07-20T02:21:00.885 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-07-20T02:21:00.885 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-07-20T02:21:00.885 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-07-20T02:21:00.886 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-07-20T02:21:00.886 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-07-20T02:21:00.886 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-07-20T02:21:00.886 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-07-20T02:21:00.886 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-07-20T02:21:00.886 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-07-20T02:21:00.886 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-07-20T02:21:00.886 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-07-20T02:21:00.887 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-07-20T02:21:00.887 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nf_conncount.ko 2023-07-20T02:21:00.887 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nf_conntrack.ko 2023-07-20T02:21:00.887 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-07-20T02:21:00.887 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-07-20T02:21:00.887 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-07-20T02:21:00.887 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nf_conntrack_h323.ko 2023-07-20T02:21:00.888 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nf_conntrack_irc.ko 2023-07-20T02:21:00.888 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-07-20T02:21:00.888 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-07-20T02:21:00.888 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-07-20T02:21:00.888 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nf_conntrack_sane.ko 2023-07-20T02:21:00.888 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nf_conntrack_sip.ko 2023-07-20T02:21:00.888 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-07-20T02:21:00.888 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-07-20T02:21:00.889 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nf_dup_netdev.ko 2023-07-20T02:21:00.889 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nf_flow_table.ko 2023-07-20T02:21:00.889 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nf_flow_table_inet.ko 2023-07-20T02:21:00.889 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nf_log_syslog.ko 2023-07-20T02:21:00.889 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nf_nat.ko 2023-07-20T02:21:00.889 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nf_nat_amanda.ko 2023-07-20T02:21:00.889 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nf_nat_ftp.ko 2023-07-20T02:21:00.889 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nf_nat_irc.ko 2023-07-20T02:21:00.890 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nf_nat_sip.ko 2023-07-20T02:21:00.890 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nf_nat_tftp.ko 2023-07-20T02:21:00.890 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nf_synproxy_core.ko 2023-07-20T02:21:00.890 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nf_tables.ko 2023-07-20T02:21:00.890 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nfnetlink.ko 2023-07-20T02:21:00.890 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nfnetlink_acct.ko 2023-07-20T02:21:00.890 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nfnetlink_log.ko 2023-07-20T02:21:00.890 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nfnetlink_osf.ko 2023-07-20T02:21:00.891 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nfnetlink_queue.ko 2023-07-20T02:21:00.891 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nft_chain_nat.ko 2023-07-20T02:21:00.891 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nft_compat.ko 2023-07-20T02:21:00.891 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nft_ct.ko 2023-07-20T02:21:00.891 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nft_dup_netdev.ko 2023-07-20T02:21:00.891 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nft_fib.ko 2023-07-20T02:21:00.891 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nft_fwd_netdev.ko 2023-07-20T02:21:00.891 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nft_hash.ko 2023-07-20T02:21:00.892 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nft_limit.ko 2023-07-20T02:21:00.892 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nft_log.ko 2023-07-20T02:21:00.892 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nft_masq.ko 2023-07-20T02:21:00.892 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nft_nat.ko 2023-07-20T02:21:00.892 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nft_numgen.ko 2023-07-20T02:21:00.892 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nft_queue.ko 2023-07-20T02:21:00.892 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nft_quota.ko 2023-07-20T02:21:00.892 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nft_redir.ko 2023-07-20T02:21:00.893 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nft_reject.ko 2023-07-20T02:21:00.893 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nft_reject_inet.ko 2023-07-20T02:21:00.893 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nft_socket.ko 2023-07-20T02:21:00.893 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/nft_tunnel.ko 2023-07-20T02:21:00.893 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_AUDIT.ko 2023-07-20T02:21:00.893 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_CHECKSUM.ko 2023-07-20T02:21:00.893 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_CLASSIFY.ko 2023-07-20T02:21:00.894 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-07-20T02:21:00.894 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_CT.ko 2023-07-20T02:21:00.894 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_DSCP.ko 2023-07-20T02:21:00.894 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_HL.ko 2023-07-20T02:21:00.894 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_HMARK.ko 2023-07-20T02:21:00.894 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_IDLETIMER.ko 2023-07-20T02:21:00.894 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_LED.ko 2023-07-20T02:21:00.894 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_LOG.ko 2023-07-20T02:21:00.895 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_MASQUERADE.ko 2023-07-20T02:21:00.895 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_NETMAP.ko 2023-07-20T02:21:00.895 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_NFLOG.ko 2023-07-20T02:21:00.895 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_NFQUEUE.ko 2023-07-20T02:21:00.895 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_RATEEST.ko 2023-07-20T02:21:00.895 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_REDIRECT.ko 2023-07-20T02:21:00.895 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_SECMARK.ko 2023-07-20T02:21:00.895 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_TCPMSS.ko 2023-07-20T02:21:00.896 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-07-20T02:21:00.896 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_TEE.ko 2023-07-20T02:21:00.896 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_TPROXY.ko 2023-07-20T02:21:00.896 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_TRACE.ko 2023-07-20T02:21:00.896 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_addrtype.ko 2023-07-20T02:21:00.896 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_bpf.ko 2023-07-20T02:21:00.896 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_cgroup.ko 2023-07-20T02:21:00.896 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_cluster.ko 2023-07-20T02:21:00.897 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_comment.ko 2023-07-20T02:21:00.897 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_connbytes.ko 2023-07-20T02:21:00.897 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_connlabel.ko 2023-07-20T02:21:00.897 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_connlimit.ko 2023-07-20T02:21:00.897 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_connmark.ko 2023-07-20T02:21:00.897 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_conntrack.ko 2023-07-20T02:21:00.897 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_cpu.ko 2023-07-20T02:21:00.897 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_dccp.ko 2023-07-20T02:21:00.898 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_devgroup.ko 2023-07-20T02:21:00.898 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_dscp.ko 2023-07-20T02:21:00.898 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_ecn.ko 2023-07-20T02:21:00.898 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_esp.ko 2023-07-20T02:21:00.898 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_hashlimit.ko 2023-07-20T02:21:00.898 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_helper.ko 2023-07-20T02:21:00.898 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_hl.ko 2023-07-20T02:21:00.898 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_ipcomp.ko 2023-07-20T02:21:00.899 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_iprange.ko 2023-07-20T02:21:00.899 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_ipvs.ko 2023-07-20T02:21:00.899 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_l2tp.ko 2023-07-20T02:21:00.899 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_length.ko 2023-07-20T02:21:00.899 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_limit.ko 2023-07-20T02:21:00.899 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_mac.ko 2023-07-20T02:21:00.899 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_mark.ko 2023-07-20T02:21:00.899 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_multiport.ko 2023-07-20T02:21:00.900 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_nat.ko 2023-07-20T02:21:00.900 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_nfacct.ko 2023-07-20T02:21:00.900 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_osf.ko 2023-07-20T02:21:00.900 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_owner.ko 2023-07-20T02:21:00.900 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_physdev.ko 2023-07-20T02:21:00.900 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_pkttype.ko 2023-07-20T02:21:00.900 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_policy.ko 2023-07-20T02:21:00.900 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_quota.ko 2023-07-20T02:21:00.901 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_rateest.ko 2023-07-20T02:21:00.901 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_realm.ko 2023-07-20T02:21:00.901 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_recent.ko 2023-07-20T02:21:00.901 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_sctp.ko 2023-07-20T02:21:00.901 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_set.ko 2023-07-20T02:21:00.901 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_socket.ko 2023-07-20T02:21:00.901 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_state.ko 2023-07-20T02:21:00.902 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_statistic.ko 2023-07-20T02:21:00.902 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_string.ko 2023-07-20T02:21:00.902 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_tcpmss.ko 2023-07-20T02:21:00.902 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_time.ko 2023-07-20T02:21:00.902 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netfilter/xt_u32.ko 2023-07-20T02:21:00.902 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netlink 2023-07-20T02:21:00.902 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/netlink/netlink_diag.ko 2023-07-20T02:21:00.902 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/nsh 2023-07-20T02:21:00.903 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/nsh/nsh.ko 2023-07-20T02:21:00.903 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/openvswitch 2023-07-20T02:21:00.903 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/openvswitch/openvswitch.ko 2023-07-20T02:21:00.903 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/openvswitch/vport-geneve.ko 2023-07-20T02:21:00.903 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/openvswitch/vport-gre.ko 2023-07-20T02:21:00.903 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/openvswitch/vport-vxlan.ko 2023-07-20T02:21:00.903 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/packet 2023-07-20T02:21:00.903 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/packet/af_packet_diag.ko 2023-07-20T02:21:00.904 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/rds 2023-07-20T02:21:00.904 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/rds/rds.ko 2023-07-20T02:21:00.904 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/rds/rds_rdma.ko 2023-07-20T02:21:00.904 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/rds/rds_tcp.ko 2023-07-20T02:21:00.904 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched 2023-07-20T02:21:00.904 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/act_csum.ko 2023-07-20T02:21:00.904 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/act_gact.ko 2023-07-20T02:21:00.904 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/act_ipt.ko 2023-07-20T02:21:00.905 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/act_mirred.ko 2023-07-20T02:21:00.905 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/act_nat.ko 2023-07-20T02:21:00.905 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/act_pedit.ko 2023-07-20T02:21:00.905 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/act_police.ko 2023-07-20T02:21:00.905 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/act_simple.ko 2023-07-20T02:21:00.905 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/act_skbedit.ko 2023-07-20T02:21:00.905 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/act_vlan.ko 2023-07-20T02:21:00.906 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/cls_basic.ko 2023-07-20T02:21:00.906 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/cls_bpf.ko 2023-07-20T02:21:00.906 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/cls_flow.ko 2023-07-20T02:21:00.906 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/cls_fw.ko 2023-07-20T02:21:00.906 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/cls_route.ko 2023-07-20T02:21:00.906 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/cls_u32.ko 2023-07-20T02:21:00.906 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/em_cmp.ko 2023-07-20T02:21:00.906 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/em_ipset.ko 2023-07-20T02:21:00.907 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/em_meta.ko 2023-07-20T02:21:00.907 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/em_nbyte.ko 2023-07-20T02:21:00.907 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/em_text.ko 2023-07-20T02:21:00.907 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/em_u32.ko 2023-07-20T02:21:00.907 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/sch_choke.ko 2023-07-20T02:21:00.907 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/sch_codel.ko 2023-07-20T02:21:00.907 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/sch_drr.ko 2023-07-20T02:21:00.907 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/sch_fq.ko 2023-07-20T02:21:00.908 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/sch_gred.ko 2023-07-20T02:21:00.908 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/sch_hfsc.ko 2023-07-20T02:21:00.908 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/sch_hhf.ko 2023-07-20T02:21:00.908 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/sch_htb.ko 2023-07-20T02:21:00.908 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/sch_ingress.ko 2023-07-20T02:21:00.908 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/sch_mqprio.ko 2023-07-20T02:21:00.908 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/sch_mqprio_lib.ko 2023-07-20T02:21:00.908 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/sch_multiq.ko 2023-07-20T02:21:00.909 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/sch_netem.ko 2023-07-20T02:21:00.909 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/sch_pie.ko 2023-07-20T02:21:00.909 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/sch_plug.ko 2023-07-20T02:21:00.909 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/sch_prio.ko 2023-07-20T02:21:00.909 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/sch_qfq.ko 2023-07-20T02:21:00.909 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/sch_red.ko 2023-07-20T02:21:00.909 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/sch_sfb.ko 2023-07-20T02:21:00.909 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/sch_sfq.ko 2023-07-20T02:21:00.910 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/sch_tbf.ko 2023-07-20T02:21:00.910 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sched/sch_teql.ko 2023-07-20T02:21:00.910 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sctp 2023-07-20T02:21:00.910 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sctp/sctp.ko 2023-07-20T02:21:00.910 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sctp/sctp_diag.ko 2023-07-20T02:21:00.910 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sunrpc 2023-07-20T02:21:00.910 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sunrpc/auth_gss 2023-07-20T02:21:00.910 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-07-20T02:21:00.911 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-07-20T02:21:00.911 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sunrpc/sunrpc.ko 2023-07-20T02:21:00.911 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sunrpc/xprtrdma 2023-07-20T02:21:00.911 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-07-20T02:21:00.911 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/tipc 2023-07-20T02:21:00.911 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/tipc/diag.ko 2023-07-20T02:21:00.911 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/tipc/tipc.ko 2023-07-20T02:21:00.911 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/unix 2023-07-20T02:21:00.912 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/unix/unix_diag.ko 2023-07-20T02:21:00.912 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/vmw_vsock 2023-07-20T02:21:00.912 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-07-20T02:21:00.912 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-07-20T02:21:00.912 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/vmw_vsock/vsock.ko 2023-07-20T02:21:00.912 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/vmw_vsock/vsock_diag.ko 2023-07-20T02:21:00.912 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/vmw_vsock/vsock_loopback.ko 2023-07-20T02:21:00.913 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/xfrm 2023-07-20T02:21:00.913 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/net/xfrm/xfrm_ipcomp.ko 2023-07-20T02:21:00.913 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/security 2023-07-20T02:21:00.913 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/security/keys 2023-07-20T02:21:00.913 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/security/keys/encrypted-keys 2023-07-20T02:21:00.913 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2023-07-20T02:21:00.913 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/security/keys/trusted-keys 2023-07-20T02:21:00.913 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/security/keys/trusted-keys/trusted.ko 2023-07-20T02:21:00.914 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/virt 2023-07-20T02:21:00.914 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/virt/lib 2023-07-20T02:21:00.914 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/kernel/virt/lib/irqbypass.ko 2023-07-20T02:21:00.914 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/modules.alias 2023-07-20T02:21:00.914 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/modules.alias.bin 2023-07-20T02:21:00.914 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/modules.builtin 2023-07-20T02:21:00.914 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/modules.builtin.bin 2023-07-20T02:21:00.914 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/modules.builtin.modinfo 2023-07-20T02:21:00.915 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/modules.dep 2023-07-20T02:21:00.915 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/modules.dep.bin 2023-07-20T02:21:00.915 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/modules.devname 2023-07-20T02:21:00.915 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/modules.order 2023-07-20T02:21:00.915 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/modules.softdep 2023-07-20T02:21:00.915 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/modules.symbols 2023-07-20T02:21:00.915 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.5.0-rc1-ga440c79eee02/modules.symbols.bin 2023-07-20T02:21:00.916 DEBUG:teuthology.task.kernel:get_image_version: 6.5.0-rc1-ga440c79eee02 2023-07-20T02:21:00.916 DEBUG:teuthology.orchestra.run.smithi007:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2023-07-20T02:21:00.944 INFO:teuthology.orchestra.run.smithi007.stdout:postinstall scriptlet (using /bin/sh): 2023-07-20T02:21:00.945 INFO:teuthology.orchestra.run.smithi007.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.5.0-rc1-ga440c79eee02 -a -r /boot/System.map-6.5.0-rc1-ga440c79eee02 ]; then 2023-07-20T02:21:00.945 INFO:teuthology.orchestra.run.smithi007.stdout:cp /boot/vmlinuz-6.5.0-rc1-ga440c79eee02 /boot/.vmlinuz-6.5.0-rc1-ga440c79eee02-rpm 2023-07-20T02:21:00.945 INFO:teuthology.orchestra.run.smithi007.stdout:cp /boot/System.map-6.5.0-rc1-ga440c79eee02 /boot/.System.map-6.5.0-rc1-ga440c79eee02-rpm 2023-07-20T02:21:00.945 INFO:teuthology.orchestra.run.smithi007.stdout:rm -f /boot/vmlinuz-6.5.0-rc1-ga440c79eee02 /boot/System.map-6.5.0-rc1-ga440c79eee02 2023-07-20T02:21:00.945 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/installkernel 6.5.0-rc1-ga440c79eee02 /boot/.vmlinuz-6.5.0-rc1-ga440c79eee02-rpm /boot/.System.map-6.5.0-rc1-ga440c79eee02-rpm 2023-07-20T02:21:00.946 INFO:teuthology.orchestra.run.smithi007.stdout:rm -f /boot/.vmlinuz-6.5.0-rc1-ga440c79eee02-rpm /boot/.System.map-6.5.0-rc1-ga440c79eee02-rpm 2023-07-20T02:21:00.946 INFO:teuthology.orchestra.run.smithi007.stdout:fi 2023-07-20T02:21:00.946 INFO:teuthology.orchestra.run.smithi007.stdout:preuninstall scriptlet (using /bin/sh): 2023-07-20T02:21:00.946 INFO:teuthology.orchestra.run.smithi007.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2023-07-20T02:21:00.946 INFO:teuthology.orchestra.run.smithi007.stdout:new-kernel-pkg --remove 6.5.0-rc1-ga440c79eee02 --rminitrd --initrdfile=/boot/initramfs-6.5.0-rc1-ga440c79eee02.img 2023-07-20T02:21:00.947 INFO:teuthology.orchestra.run.smithi007.stdout:elif [ -x /usr/bin/kernel-install ]; then 2023-07-20T02:21:00.947 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-install remove 6.5.0-rc1-ga440c79eee02 2023-07-20T02:21:00.947 INFO:teuthology.orchestra.run.smithi007.stdout:fi 2023-07-20T02:21:00.947 INFO:teuthology.orchestra.run.smithi007.stdout:postuninstall scriptlet (using /bin/sh): 2023-07-20T02:21:00.947 INFO:teuthology.orchestra.run.smithi007.stdout:if [ -x /sbin/update-bootloader ]; then 2023-07-20T02:21:00.947 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/update-bootloader --remove 6.5.0-rc1-ga440c79eee02 2023-07-20T02:21:00.947 INFO:teuthology.orchestra.run.smithi007.stdout:fi 2023-07-20T02:21:00.948 DEBUG:teuthology.orchestra.run.smithi007:> sudo rpm -qi grub2-tools 2023-07-20T02:21:01.029 INFO:teuthology.orchestra.run.smithi007.stdout:Name : grub2-tools 2023-07-20T02:21:01.029 INFO:teuthology.orchestra.run.smithi007.stdout:Epoch : 1 2023-07-20T02:21:01.029 INFO:teuthology.orchestra.run.smithi007.stdout:Version : 2.02 2023-07-20T02:21:01.030 INFO:teuthology.orchestra.run.smithi007.stdout:Release : 123.el8 2023-07-20T02:21:01.030 INFO:teuthology.orchestra.run.smithi007.stdout:Architecture: x86_64 2023-07-20T02:21:01.030 INFO:teuthology.orchestra.run.smithi007.stdout:Install Date: Thu 14 Jul 2022 06:22:19 PM UTC 2023-07-20T02:21:01.030 INFO:teuthology.orchestra.run.smithi007.stdout:Group : System Environment/Base 2023-07-20T02:21:01.030 INFO:teuthology.orchestra.run.smithi007.stdout:Size : 9265450 2023-07-20T02:21:01.030 INFO:teuthology.orchestra.run.smithi007.stdout:License : GPLv3+ 2023-07-20T02:21:01.030 INFO:teuthology.orchestra.run.smithi007.stdout:Signature : RSA/SHA256, Thu 31 Mar 2022 07:24:35 PM UTC, Key ID 199e2f91fd431d51 2023-07-20T02:21:01.031 INFO:teuthology.orchestra.run.smithi007.stdout:Source RPM : grub2-2.02-123.el8.src.rpm 2023-07-20T02:21:01.031 INFO:teuthology.orchestra.run.smithi007.stdout:Build Date : Mon 28 Mar 2022 02:12:09 PM UTC 2023-07-20T02:21:01.031 INFO:teuthology.orchestra.run.smithi007.stdout:Build Host : x86-vm-09.build.eng.bos.redhat.com 2023-07-20T02:21:01.031 INFO:teuthology.orchestra.run.smithi007.stdout:Relocations : (not relocatable) 2023-07-20T02:21:01.031 INFO:teuthology.orchestra.run.smithi007.stdout:Packager : Red Hat, Inc. 2023-07-20T02:21:01.031 INFO:teuthology.orchestra.run.smithi007.stdout:Vendor : Red Hat, Inc. 2023-07-20T02:21:01.031 INFO:teuthology.orchestra.run.smithi007.stdout:URL : http://www.gnu.org/software/grub/ 2023-07-20T02:21:01.032 INFO:teuthology.orchestra.run.smithi007.stdout:Summary : Support tools for GRUB. 2023-07-20T02:21:01.032 INFO:teuthology.orchestra.run.smithi007.stdout:Description : 2023-07-20T02:21:01.032 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:21:01.032 INFO:teuthology.orchestra.run.smithi007.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-07-20T02:21:01.032 INFO:teuthology.orchestra.run.smithi007.stdout:customizable bootloader with modular architecture. It supports a rich 2023-07-20T02:21:01.032 INFO:teuthology.orchestra.run.smithi007.stdout:variety of kernel formats, file systems, computer architectures and 2023-07-20T02:21:01.032 INFO:teuthology.orchestra.run.smithi007.stdout:hardware devices. 2023-07-20T02:21:01.033 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:21:01.033 INFO:teuthology.orchestra.run.smithi007.stdout:This subpackage provides tools for support of all platforms. 2023-07-20T02:21:01.034 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-07-20T02:21:01.034 INFO:teuthology.task.kernel:Updating grub on smithi007 to boot 6.5.0-rc1-ga440c79eee02 2023-07-20T02:21:01.034 DEBUG:teuthology.orchestra.run.smithi007:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-07-20T02:21:04.922 INFO:teuthology.orchestra.run.smithi007.stderr:Generating grub configuration file ... 2023-07-20T02:21:06.674 INFO:teuthology.orchestra.run.smithi007.stderr:done 2023-07-20T02:21:06.676 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2023-07-20T02:21:06.694 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.0QYYhLGWix 2023-07-20T02:21:06.695 DEBUG:teuthology.orchestra.run.smithi007:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.0QYYhLGWix 2023-07-20T02:21:06.768 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.0QYYhLGWix 2023-07-20T02:21:06.917 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.0QYYhLGWix is 6KB 2023-07-20T02:21:06.938 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.0QYYhLGWix 2023-07-20T02:21:06.954 DEBUG:teuthology.orchestra.run.smithi007:> sudo /bin/ls /boot/loader/entries || true 2023-07-20T02:21:07.020 INFO:teuthology.orchestra.run.smithi007.stdout:5a84457b7b52416e95d47d02e2e4a60f-0-rescue.conf 2023-07-20T02:21:07.020 INFO:teuthology.orchestra.run.smithi007.stdout:5a84457b7b52416e95d47d02e2e4a60f-4.18.0-372.9.1.el8.x86_64.conf 2023-07-20T02:21:07.021 INFO:teuthology.orchestra.run.smithi007.stdout:5a84457b7b52416e95d47d02e2e4a60f-6.5.0-rc1-ga440c79eee02.conf 2023-07-20T02:21:07.022 DEBUG:teuthology.orchestra.run.smithi007:> sudo grub2-set-default 5a84457b7b52416e95d47d02e2e4a60f-6.5.0-rc1-ga440c79eee02 2023-07-20T02:21:07.256 DEBUG:teuthology.orchestra.run.smithi007:> sudo shutdown -r now 2023-07-20T02:21:37.259 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-07-20T02:21:37.259 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-07-20T02:21:37.260 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T02:21:55.696 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.7 2023-07-20T02:22:04.698 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-07-20T02:22:04.698 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T02:23:04.700 DEBUG:teuthology.orchestra.remote:timed out 2023-07-20T02:23:16.702 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-07-20T02:23:16.703 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T02:23:17.078 DEBUG:teuthology.orchestra.run.smithi007:> true 2023-07-20T02:23:17.473 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-07-20T02:23:17.474 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.5.0-rc1-ga440c79eee02"... 2023-07-20T02:23:17.474 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2023-07-20T02:23:17.517 INFO:teuthology.orchestra.run.smithi007.stdout:6.5.0-rc1-ga440c79eee02 2023-07-20T02:23:17.517 DEBUG:teuthology.task.kernel:current kernel version is 6.5.0-rc1-ga440c79eee02 vs 6.5.0-rc1-ga440c79eee02 2023-07-20T02:23:17.517 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-07-20T02:23:17.517 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-07-20T02:23:18.518 DEBUG:teuthology.parallel:result is None 2023-07-20T02:23:18.519 INFO:teuthology.run_tasks:Running task internal.base... 2023-07-20T02:23:18.532 INFO:teuthology.task.internal:Creating test directory... 2023-07-20T02:23:18.533 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-20T02:23:18.537 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-20T02:23:18.541 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-20T02:23:18.580 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-07-20T02:23:18.590 INFO:teuthology.run_tasks:Running task internal.archive... 2023-07-20T02:23:18.599 INFO:teuthology.task.internal:Creating archive directory... 2023-07-20T02:23:18.599 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-20T02:23:18.625 DEBUG:teuthology.orchestra.run.smithi158:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-20T02:23:18.627 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-20T02:23:18.682 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-07-20T02:23:18.691 INFO:teuthology.task.internal:Enabling coredump saving... 2023-07-20T02:23:18.691 DEBUG:teuthology.orchestra.run.smithi158:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-20T02:23:18.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T02:23:18.727 DEBUG:teuthology.orchestra.run.smithi188:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-20T02:23:18.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T02:23:18.746 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-20T02:23:18.750 DEBUG:teuthology.orchestra.run.smithi158:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-20T02:23:18.769 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-20T02:23:18.824 INFO:teuthology.orchestra.run.smithi158.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T02:23:18.833 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T02:23:18.842 INFO:teuthology.orchestra.run.smithi158.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T02:23:18.891 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T02:23:18.902 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T02:23:18.923 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T02:23:18.926 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-07-20T02:23:18.933 INFO:teuthology.task.internal:Configuring sudo... 2023-07-20T02:23:18.933 DEBUG:teuthology.orchestra.run.smithi007:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-20T02:23:18.938 DEBUG:teuthology.orchestra.run.smithi158:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-20T02:23:18.940 DEBUG:teuthology.orchestra.run.smithi188:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-20T02:23:19.003 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-07-20T02:23:19.012 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-07-20T02:23:19.013 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-20T02:23:19.017 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-20T02:23:19.019 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-20T02:23:19.055 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T02:23:19.168 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T02:23:19.273 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T02:23:19.347 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T02:23:19.426 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-07-20T02:23:19.427 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-20T02:23:19.495 DEBUG:teuthology.orchestra.run.smithi158:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T02:23:19.550 DEBUG:teuthology.orchestra.run.smithi158:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T02:23:19.599 DEBUG:teuthology.orchestra.run.smithi158:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T02:23:19.688 DEBUG:teuthology.orchestra.run.smithi158:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T02:23:19.727 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-07-20T02:23:19.727 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-20T02:23:19.795 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T02:23:19.846 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T02:23:19.902 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T02:23:19.952 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T02:23:19.991 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2023-07-20T02:23:19.991 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-20T02:23:20.110 DEBUG:teuthology.orchestra.run.smithi007:> sudo service rsyslog restart 2023-07-20T02:23:20.115 DEBUG:teuthology.orchestra.run.smithi158:> sudo service rsyslog restart 2023-07-20T02:23:20.117 DEBUG:teuthology.orchestra.run.smithi188:> sudo service rsyslog restart 2023-07-20T02:23:20.180 INFO:teuthology.orchestra.run.smithi188.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T02:23:20.232 INFO:teuthology.orchestra.run.smithi158.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T02:23:20.233 INFO:teuthology.orchestra.run.smithi007.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T02:23:20.695 INFO:teuthology.run_tasks:Running task internal.timer... 2023-07-20T02:23:20.706 INFO:teuthology.task.internal:Starting timer... 2023-07-20T02:23:20.706 INFO:teuthology.run_tasks:Running task pcp... 2023-07-20T02:23:20.723 INFO:teuthology.run_tasks:Running task selinux... 2023-07-20T02:23:20.739 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-07-20T02:23:20.740 DEBUG:teuthology.orchestra.run.smithi007:> sudo service auditd rotate 2023-07-20T02:23:20.854 INFO:teuthology.orchestra.run.smithi007.stdout:Rotating logs: 2023-07-20T02:23:20.857 DEBUG:teuthology.orchestra.run.smithi158:> sudo service auditd rotate 2023-07-20T02:23:20.919 INFO:teuthology.orchestra.run.smithi158.stdout:Rotating logs: 2023-07-20T02:23:20.921 DEBUG:teuthology.orchestra.run.smithi188:> sudo service auditd rotate 2023-07-20T02:23:20.982 INFO:teuthology.orchestra.run.smithi188.stdout:Rotating logs: 2023-07-20T02:23:20.984 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-07-20T02:23:20.985 DEBUG:teuthology.orchestra.run.smithi007:> /usr/sbin/getenforce 2023-07-20T02:23:21.036 INFO:teuthology.orchestra.run.smithi007.stdout:Permissive 2023-07-20T02:23:21.036 DEBUG:teuthology.orchestra.run.smithi158:> /usr/sbin/getenforce 2023-07-20T02:23:21.063 INFO:teuthology.orchestra.run.smithi158.stdout:Permissive 2023-07-20T02:23:21.064 DEBUG:teuthology.orchestra.run.smithi188:> /usr/sbin/getenforce 2023-07-20T02:23:21.087 INFO:teuthology.orchestra.run.smithi188.stdout:Permissive 2023-07-20T02:23:21.088 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi007.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi158.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi188.front.sepia.ceph.com': 'permissive'} 2023-07-20T02:23:21.088 DEBUG:teuthology.orchestra.run.smithi007:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T02:23:21.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T02:23:21.159 DEBUG:teuthology.orchestra.run.smithi158:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T02:23:21.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T02:23:21.187 DEBUG:teuthology.orchestra.run.smithi188:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T02:23:21.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T02:23:21.214 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-07-20T02:23:21.215 DEBUG:teuthology.orchestra.run.smithi007:> sudo /usr/sbin/setenforce permissive 2023-07-20T02:23:21.285 DEBUG:teuthology.orchestra.run.smithi158:> sudo /usr/sbin/setenforce permissive 2023-07-20T02:23:21.313 DEBUG:teuthology.orchestra.run.smithi188:> sudo /usr/sbin/setenforce permissive 2023-07-20T02:23:21.340 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-07-20T02:23:21.353 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-07-20T02:23:21.354 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-07-20T02:23:21.382 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-07-20T02:23:21.383 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi007.front.sepia.ceph.com,smithi158.front.sepia.ceph.com,smithi188.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-07-20T02:38:26.253 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi007.front.sepia.ceph.com'), Remote(name='ubuntu@smithi158.front.sepia.ceph.com'), Remote(name='ubuntu@smithi188.front.sepia.ceph.com')] 2023-07-20T02:38:26.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-07-20T02:38:26.257 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T02:38:26.348 DEBUG:teuthology.orchestra.run.smithi007:> true 2023-07-20T02:38:26.464 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-07-20T02:38:26.465 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2023-07-20T02:38:26.465 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T02:38:26.533 DEBUG:teuthology.orchestra.run.smithi158:> true 2023-07-20T02:38:26.606 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi158.front.sepia.ceph.com' 2023-07-20T02:38:26.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2023-07-20T02:38:26.607 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T02:38:26.675 DEBUG:teuthology.orchestra.run.smithi188:> true 2023-07-20T02:38:26.742 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi188.front.sepia.ceph.com' 2023-07-20T02:38:26.743 INFO:teuthology.run_tasks:Running task clock... 2023-07-20T02:38:26.756 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-07-20T02:38:26.757 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-20T02:38:26.757 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T02:38:26.762 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-20T02:38:26.763 DEBUG:teuthology.orchestra.run.smithi158:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T02:38:26.765 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-20T02:38:26.765 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T02:38:26.797 INFO:teuthology.orchestra.run.smithi158.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-20T02:38:26.812 INFO:teuthology.orchestra.run.smithi158.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-20T02:38:26.817 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-20T02:38:26.832 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-20T02:38:26.837 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-20T02:38:26.874 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-20T02:38:26.892 INFO:teuthology.orchestra.run.smithi158.stderr:sudo: ntpd: command not found 2023-07-20T02:38:26.904 INFO:teuthology.orchestra.run.smithi158.stdout:506 Cannot talk to daemon 2023-07-20T02:38:26.910 INFO:teuthology.orchestra.run.smithi188.stderr:sudo: ntpd: command not found 2023-07-20T02:38:26.919 INFO:teuthology.orchestra.run.smithi158.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-20T02:38:26.922 INFO:teuthology.orchestra.run.smithi188.stdout:506 Cannot talk to daemon 2023-07-20T02:38:26.933 INFO:teuthology.orchestra.run.smithi158.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-20T02:38:26.936 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-20T02:38:26.950 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-20T02:38:26.975 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: ntpd: command not found 2023-07-20T02:38:27.004 INFO:teuthology.orchestra.run.smithi158.stderr:bash: ntpq: command not found 2023-07-20T02:38:27.005 INFO:teuthology.orchestra.run.smithi007.stdout:506 Cannot talk to daemon 2023-07-20T02:38:27.007 INFO:teuthology.orchestra.run.smithi158.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T02:38:27.007 INFO:teuthology.orchestra.run.smithi158.stdout:=============================================================================== 2023-07-20T02:38:27.008 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T02:38:27.008 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T02:38:27.008 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T02:38:27.008 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T02:38:27.025 INFO:teuthology.orchestra.run.smithi188.stderr:bash: ntpq: command not found 2023-07-20T02:38:27.028 INFO:teuthology.orchestra.run.smithi188.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T02:38:27.028 INFO:teuthology.orchestra.run.smithi188.stdout:=============================================================================== 2023-07-20T02:38:27.028 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T02:38:27.029 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T02:38:27.029 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T02:38:27.029 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T02:38:27.041 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-20T02:38:27.076 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-20T02:38:27.163 INFO:teuthology.orchestra.run.smithi007.stderr:bash: ntpq: command not found 2023-07-20T02:38:27.170 INFO:teuthology.orchestra.run.smithi007.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T02:38:27.171 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================== 2023-07-20T02:38:27.171 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T02:38:27.171 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T02:38:27.171 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T02:38:27.171 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T02:38:27.172 INFO:teuthology.run_tasks:Running task pexec... 2023-07-20T02:38:27.182 INFO:teuthology.task.pexec:Executing custom commands... 2023-07-20T02:38:27.182 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi007.front.sepia.ceph.com 2023-07-20T02:38:27.183 DEBUG:teuthology.orchestra.run.smithi007:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-20T02:38:27.183 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi158.front.sepia.ceph.com 2023-07-20T02:38:27.183 DEBUG:teuthology.orchestra.run.smithi158:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-20T02:38:27.184 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi188.front.sepia.ceph.com 2023-07-20T02:38:27.184 DEBUG:teuthology.orchestra.run.smithi188:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-20T02:38:27.556 INFO:teuthology.orchestra.run.smithi188.stdout:Updating Subscription Management repositories. 2023-07-20T02:38:27.558 INFO:teuthology.orchestra.run.smithi158.stdout:Updating Subscription Management repositories. 2023-07-20T02:38:27.698 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-07-20T02:38:28.713 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:07:22 ago on Thu 20 Jul 2023 02:31:06 AM UTC. 2023-07-20T02:38:28.738 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:07:23 ago on Thu 20 Jul 2023 02:31:05 AM UTC. 2023-07-20T02:38:29.033 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:06 ago on Thu 20 Jul 2023 02:36:22 AM UTC. 2023-07-20T02:38:30.424 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2023-07-20T02:38:30.425 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-07-20T02:38:30.425 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2023-07-20T02:38:30.425 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-07-20T02:38:30.426 INFO:teuthology.orchestra.run.smithi188.stdout:Resetting modules: 2023-07-20T02:38:30.426 INFO:teuthology.orchestra.run.smithi188.stdout: container-tools 2023-07-20T02:38:30.426 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-07-20T02:38:30.426 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2023-07-20T02:38:30.427 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2023-07-20T02:38:30.427 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-07-20T02:38:30.474 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2023-07-20T02:38:30.476 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-07-20T02:38:30.476 INFO:teuthology.orchestra.run.smithi158.stdout: Package Architecture Version Repository Size 2023-07-20T02:38:30.476 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-07-20T02:38:30.476 INFO:teuthology.orchestra.run.smithi158.stdout:Resetting modules: 2023-07-20T02:38:30.477 INFO:teuthology.orchestra.run.smithi158.stdout: container-tools 2023-07-20T02:38:30.477 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-07-20T02:38:30.477 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2023-07-20T02:38:30.477 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-07-20T02:38:30.477 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-07-20T02:38:31.097 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-07-20T02:38:31.099 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-07-20T02:38:31.099 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2023-07-20T02:38:31.099 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-07-20T02:38:31.099 INFO:teuthology.orchestra.run.smithi007.stdout:Resetting modules: 2023-07-20T02:38:31.099 INFO:teuthology.orchestra.run.smithi007.stdout: container-tools 2023-07-20T02:38:31.099 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:38:31.100 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-07-20T02:38:31.100 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-07-20T02:38:31.100 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:38:31.404 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:07:24 ago on Thu 20 Jul 2023 02:31:06 AM UTC. 2023-07-20T02:38:31.547 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:07:26 ago on Thu 20 Jul 2023 02:31:05 AM UTC. 2023-07-20T02:38:32.365 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:09 ago on Thu 20 Jul 2023 02:36:22 AM UTC. 2023-07-20T02:38:36.555 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2023-07-20T02:38:37.027 INFO:teuthology.orchestra.run.smithi188.stdout:Updating Subscription Management repositories. 2023-07-20T02:38:37.099 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2023-07-20T02:38:37.576 INFO:teuthology.orchestra.run.smithi158.stdout:Updating Subscription Management repositories. 2023-07-20T02:38:38.032 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:07:31 ago on Thu 20 Jul 2023 02:31:06 AM UTC. 2023-07-20T02:38:38.105 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-07-20T02:38:38.667 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:07:33 ago on Thu 20 Jul 2023 02:31:05 AM UTC. 2023-07-20T02:38:38.744 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-07-20T02:38:39.893 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2023-07-20T02:38:39.984 INFO:teuthology.orchestra.run.smithi188.stdout:============================================================================================================ 2023-07-20T02:38:39.985 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2023-07-20T02:38:39.985 INFO:teuthology.orchestra.run.smithi188.stdout:============================================================================================================ 2023-07-20T02:38:39.985 INFO:teuthology.orchestra.run.smithi188.stdout:Installing group/module packages: 2023-07-20T02:38:39.985 INFO:teuthology.orchestra.run.smithi188.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-20T02:38:39.985 INFO:teuthology.orchestra.run.smithi188.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-20T02:38:39.985 INFO:teuthology.orchestra.run.smithi188.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-20T02:38:39.986 INFO:teuthology.orchestra.run.smithi188.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-20T02:38:39.986 INFO:teuthology.orchestra.run.smithi188.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-20T02:38:39.986 INFO:teuthology.orchestra.run.smithi188.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-20T02:38:39.986 INFO:teuthology.orchestra.run.smithi188.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-20T02:38:39.986 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2023-07-20T02:38:39.986 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-20T02:38:39.986 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-20T02:38:39.986 INFO:teuthology.orchestra.run.smithi188.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-20T02:38:39.987 INFO:teuthology.orchestra.run.smithi188.stdout:Installing module profiles: 2023-07-20T02:38:39.987 INFO:teuthology.orchestra.run.smithi188.stdout: container-tools/common 2023-07-20T02:38:39.987 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-07-20T02:38:39.987 INFO:teuthology.orchestra.run.smithi188.stdout:Enabling module streams: 2023-07-20T02:38:39.987 INFO:teuthology.orchestra.run.smithi188.stdout: container-tools rhel8 2023-07-20T02:38:39.987 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-07-20T02:38:39.987 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2023-07-20T02:38:39.987 INFO:teuthology.orchestra.run.smithi188.stdout:============================================================================================================ 2023-07-20T02:38:39.988 INFO:teuthology.orchestra.run.smithi188.stdout:Install 10 Packages 2023-07-20T02:38:39.988 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-07-20T02:38:39.988 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 18 M 2023-07-20T02:38:39.989 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 65 M 2023-07-20T02:38:39.989 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2023-07-20T02:38:40.395 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2023-07-20T02:38:40.442 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:17 ago on Thu 20 Jul 2023 02:36:22 AM UTC. 2023-07-20T02:38:40.462 INFO:teuthology.orchestra.run.smithi188.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 198 kB/s | 94 kB 00:00 2023-07-20T02:38:40.487 INFO:teuthology.orchestra.run.smithi158.stdout:============================================================================================================ 2023-07-20T02:38:40.487 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repository Size 2023-07-20T02:38:40.488 INFO:teuthology.orchestra.run.smithi158.stdout:============================================================================================================ 2023-07-20T02:38:40.488 INFO:teuthology.orchestra.run.smithi158.stdout:Installing group/module packages: 2023-07-20T02:38:40.488 INFO:teuthology.orchestra.run.smithi158.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-20T02:38:40.488 INFO:teuthology.orchestra.run.smithi158.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-20T02:38:40.488 INFO:teuthology.orchestra.run.smithi158.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-20T02:38:40.488 INFO:teuthology.orchestra.run.smithi158.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-20T02:38:40.488 INFO:teuthology.orchestra.run.smithi158.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-20T02:38:40.489 INFO:teuthology.orchestra.run.smithi158.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-20T02:38:40.489 INFO:teuthology.orchestra.run.smithi158.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-20T02:38:40.489 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2023-07-20T02:38:40.489 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-20T02:38:40.489 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-20T02:38:40.489 INFO:teuthology.orchestra.run.smithi158.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-20T02:38:40.489 INFO:teuthology.orchestra.run.smithi158.stdout:Installing module profiles: 2023-07-20T02:38:40.489 INFO:teuthology.orchestra.run.smithi158.stdout: container-tools/common 2023-07-20T02:38:40.490 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-07-20T02:38:40.490 INFO:teuthology.orchestra.run.smithi158.stdout:Enabling module streams: 2023-07-20T02:38:40.490 INFO:teuthology.orchestra.run.smithi158.stdout: container-tools rhel8 2023-07-20T02:38:40.490 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-07-20T02:38:40.490 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2023-07-20T02:38:40.490 INFO:teuthology.orchestra.run.smithi158.stdout:============================================================================================================ 2023-07-20T02:38:40.490 INFO:teuthology.orchestra.run.smithi158.stdout:Install 10 Packages 2023-07-20T02:38:40.491 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-07-20T02:38:40.491 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 18 M 2023-07-20T02:38:40.491 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 65 M 2023-07-20T02:38:40.491 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2023-07-20T02:38:40.513 INFO:teuthology.orchestra.run.smithi188.stdout:(2/10): yajl-2.1.0-10.el8.x86_64.rpm 78 kB/s | 41 kB 00:00 2023-07-20T02:38:40.537 INFO:teuthology.orchestra.run.smithi188.stdout:(3/10): python3-pytoml-0.1.14-5.git7dea353.el8. 45 kB/s | 25 kB 00:00 2023-07-20T02:38:40.700 INFO:teuthology.orchestra.run.smithi158.stdout:(1/10): python3-pytoml-0.1.14-5.git7dea353.el8. 118 kB/s | 25 kB 00:00 2023-07-20T02:38:40.712 INFO:teuthology.orchestra.run.smithi188.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 197 kB/s | 49 kB 00:00 2023-07-20T02:38:40.765 INFO:teuthology.orchestra.run.smithi188.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 2.0 MB/s | 524 kB 00:00 2023-07-20T02:38:40.776 INFO:teuthology.orchestra.run.smithi158.stdout:(2/10): python3-pyxdg-0.25-16.el8.noarch.rpm 329 kB/s | 94 kB 00:00 2023-07-20T02:38:40.801 INFO:teuthology.orchestra.run.smithi158.stdout:(3/10): yajl-2.1.0-10.el8.x86_64.rpm 131 kB/s | 41 kB 00:00 2023-07-20T02:38:40.826 INFO:teuthology.orchestra.run.smithi158.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 393 kB/s | 49 kB 00:00 2023-07-20T02:38:40.918 INFO:teuthology.orchestra.run.smithi158.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 3.6 MB/s | 524 kB 00:00 2023-07-20T02:38:40.921 INFO:teuthology.orchestra.run.smithi188.stdout:(6/10): buildah-1.26.2-1.module+el8.6.0+16771+2 21 MB/s | 8.0 MB 00:00 2023-07-20T02:38:40.946 INFO:teuthology.orchestra.run.smithi188.stdout:(7/10): python3-podman-4.0.1-1.module+el8.6.0+1 639 kB/s | 149 kB 00:00 2023-07-20T02:38:40.989 INFO:teuthology.orchestra.run.smithi188.stdout:(8/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 944 kB/s | 213 kB 00:00 2023-07-20T02:38:41.001 INFO:teuthology.orchestra.run.smithi158.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 856 kB/s | 149 kB 00:00 2023-07-20T02:38:41.044 INFO:teuthology.orchestra.run.smithi158.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 1.7 MB/s | 213 kB 00:00 2023-07-20T02:38:41.144 INFO:teuthology.orchestra.run.smithi158.stdout:(8/10): buildah-1.26.2-1.module+el8.6.0+16771+2 23 MB/s | 8.0 MB 00:00 2023-07-20T02:38:41.248 INFO:teuthology.orchestra.run.smithi188.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 22 MB/s | 7.0 MB 00:00 2023-07-20T02:38:41.297 INFO:teuthology.orchestra.run.smithi188.stdout:(10/10): toolbox-0.0.99.3-0.6.module+el8.6.0+16 6.3 MB/s | 2.2 MB 00:00 2023-07-20T02:38:41.298 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2023-07-20T02:38:41.298 INFO:teuthology.orchestra.run.smithi188.stdout:Total 14 MB/s | 18 MB 00:01 2023-07-20T02:38:41.319 INFO:teuthology.orchestra.run.smithi158.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 22 MB/s | 7.0 MB 00:00 2023-07-20T02:38:41.370 INFO:teuthology.orchestra.run.smithi158.stdout:(10/10): toolbox-0.0.99.3-0.6.module+el8.6.0+16 6.7 MB/s | 2.2 MB 00:00 2023-07-20T02:38:41.370 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2023-07-20T02:38:41.371 INFO:teuthology.orchestra.run.smithi158.stdout:Total 21 MB/s | 18 MB 00:00 2023-07-20T02:38:41.500 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2023-07-20T02:38:41.518 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2023-07-20T02:38:41.519 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2023-07-20T02:38:41.568 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2023-07-20T02:38:41.587 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2023-07-20T02:38:41.587 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2023-07-20T02:38:41.700 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2023-07-20T02:38:41.704 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2023-07-20T02:38:41.770 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2023-07-20T02:38:41.774 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2023-07-20T02:38:42.202 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2023-07-20T02:38:42.350 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2023-07-20T02:38:42.389 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-20T02:38:42.474 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-07-20T02:38:42.560 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T02:38:42.580 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================================================ 2023-07-20T02:38:42.580 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2023-07-20T02:38:42.580 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================================================ 2023-07-20T02:38:42.580 INFO:teuthology.orchestra.run.smithi007.stdout:Installing group/module packages: 2023-07-20T02:38:42.580 INFO:teuthology.orchestra.run.smithi007.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-20T02:38:42.580 INFO:teuthology.orchestra.run.smithi007.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-20T02:38:42.581 INFO:teuthology.orchestra.run.smithi007.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-20T02:38:42.581 INFO:teuthology.orchestra.run.smithi007.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-20T02:38:42.581 INFO:teuthology.orchestra.run.smithi007.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-20T02:38:42.581 INFO:teuthology.orchestra.run.smithi007.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-20T02:38:42.581 INFO:teuthology.orchestra.run.smithi007.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-20T02:38:42.581 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2023-07-20T02:38:42.581 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-20T02:38:42.582 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-20T02:38:42.582 INFO:teuthology.orchestra.run.smithi007.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-20T02:38:42.582 INFO:teuthology.orchestra.run.smithi007.stdout:Installing module profiles: 2023-07-20T02:38:42.582 INFO:teuthology.orchestra.run.smithi007.stdout: container-tools/common 2023-07-20T02:38:42.582 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:38:42.582 INFO:teuthology.orchestra.run.smithi007.stdout:Enabling module streams: 2023-07-20T02:38:42.582 INFO:teuthology.orchestra.run.smithi007.stdout: container-tools rhel8 2023-07-20T02:38:42.582 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:38:42.583 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-07-20T02:38:42.583 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================================================ 2023-07-20T02:38:42.583 INFO:teuthology.orchestra.run.smithi007.stdout:Install 10 Packages 2023-07-20T02:38:42.583 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:38:42.584 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 18 M 2023-07-20T02:38:42.584 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 65 M 2023-07-20T02:38:42.584 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-07-20T02:38:42.592 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-20T02:38:42.734 INFO:teuthology.orchestra.run.smithi007.stdout:(1/10): python3-pytoml-0.1.14-5.git7dea353.el8. 165 kB/s | 25 kB 00:00 2023-07-20T02:38:42.780 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-20T02:38:42.797 INFO:teuthology.orchestra.run.smithi007.stdout:(2/10): python3-pyxdg-0.25-16.el8.noarch.rpm 448 kB/s | 94 kB 00:00 2023-07-20T02:38:42.818 INFO:teuthology.orchestra.run.smithi007.stdout:(3/10): yajl-2.1.0-10.el8.x86_64.rpm 174 kB/s | 41 kB 00:00 2023-07-20T02:38:42.818 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T02:38:42.926 INFO:teuthology.orchestra.run.smithi007.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 257 kB/s | 49 kB 00:00 2023-07-20T02:38:42.960 INFO:teuthology.orchestra.run.smithi007.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 3.1 MB/s | 524 kB 00:00 2023-07-20T02:38:42.973 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-20T02:38:43.013 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-20T02:38:43.088 INFO:teuthology.orchestra.run.smithi007.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 929 kB/s | 149 kB 00:00 2023-07-20T02:38:43.127 INFO:teuthology.orchestra.run.smithi007.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 1.2 MB/s | 213 kB 00:00 2023-07-20T02:38:43.247 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-20T02:38:43.286 INFO:teuthology.orchestra.run.smithi007.stdout:(8/10): buildah-1.26.2-1.module+el8.6.0+16771+2 17 MB/s | 8.0 MB 00:00 2023-07-20T02:38:43.354 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-20T02:38:43.378 INFO:teuthology.orchestra.run.smithi007.stdout:(9/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 8.8 MB/s | 2.2 MB 00:00 2023-07-20T02:38:43.511 INFO:teuthology.orchestra.run.smithi007.stdout:(10/10): skopeo-1.9.1-1.module+el8.6.0+16771+28 17 MB/s | 7.0 MB 00:00 2023-07-20T02:38:43.512 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-07-20T02:38:43.513 INFO:teuthology.orchestra.run.smithi007.stdout:Total 20 MB/s | 18 MB 00:00 2023-07-20T02:38:43.623 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-20T02:38:43.852 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-07-20T02:38:43.877 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-07-20T02:38:43.877 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-07-20T02:38:44.133 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-20T02:38:44.417 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-07-20T02:38:44.422 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-07-20T02:38:44.429 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-20T02:38:44.993 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-07-20T02:38:45.012 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-20T02:38:45.195 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-20T02:38:45.208 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-20T02:38:45.342 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-20T02:38:45.417 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-20T02:38:45.445 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T02:38:45.592 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-20T02:38:45.594 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T02:38:45.667 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-20T02:38:45.759 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-20T02:38:45.843 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T02:38:45.844 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-20T02:38:45.844 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T02:38:45.844 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-20T02:38:45.844 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-20T02:38:45.844 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-20T02:38:45.844 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-20T02:38:45.844 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-20T02:38:45.845 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-20T02:38:45.845 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-20T02:38:45.882 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-20T02:38:45.994 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T02:38:46.285 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-20T02:38:46.303 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T02:38:46.303 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-20T02:38:46.303 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T02:38:46.304 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-20T02:38:46.304 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-20T02:38:46.304 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-20T02:38:46.304 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-20T02:38:46.304 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-20T02:38:46.305 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-20T02:38:46.305 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-20T02:38:47.072 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-20T02:38:47.072 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:07:40 ago on Thu 20 Jul 2023 02:31:06 AM UTC. 2023-07-20T02:38:47.151 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-20T02:38:47.553 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-20T02:38:47.554 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:07:42 ago on Thu 20 Jul 2023 02:31:05 AM UTC. 2023-07-20T02:38:48.131 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-20T02:38:48.436 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-20T02:38:48.625 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-20T02:38:48.857 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T02:38:49.246 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T02:38:49.246 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-20T02:38:49.247 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T02:38:49.247 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-20T02:38:49.247 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-20T02:38:49.247 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-20T02:38:49.247 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-20T02:38:49.247 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-20T02:38:49.247 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-20T02:38:49.248 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-20T02:38:50.763 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-20T02:38:50.764 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:28 ago on Thu 20 Jul 2023 02:36:22 AM UTC. 2023-07-20T02:38:51.923 INFO:teuthology.orchestra.run.smithi188.stdout:Installed products updated. 2023-07-20T02:38:52.121 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-07-20T02:38:52.121 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2023-07-20T02:38:52.122 INFO:teuthology.orchestra.run.smithi188.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T02:38:52.122 INFO:teuthology.orchestra.run.smithi188.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T02:38:52.122 INFO:teuthology.orchestra.run.smithi188.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T02:38:52.123 INFO:teuthology.orchestra.run.smithi188.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-20T02:38:52.123 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-20T02:38:52.123 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-20T02:38:52.124 INFO:teuthology.orchestra.run.smithi188.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T02:38:52.124 INFO:teuthology.orchestra.run.smithi188.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T02:38:52.124 INFO:teuthology.orchestra.run.smithi188.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T02:38:52.124 INFO:teuthology.orchestra.run.smithi188.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-20T02:38:52.125 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-07-20T02:38:52.125 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2023-07-20T02:38:52.299 DEBUG:teuthology.parallel:result is None 2023-07-20T02:38:52.431 INFO:teuthology.orchestra.run.smithi158.stdout:Installed products updated. 2023-07-20T02:38:52.690 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-07-20T02:38:52.691 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2023-07-20T02:38:52.691 INFO:teuthology.orchestra.run.smithi158.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T02:38:52.691 INFO:teuthology.orchestra.run.smithi158.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T02:38:52.691 INFO:teuthology.orchestra.run.smithi158.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T02:38:52.691 INFO:teuthology.orchestra.run.smithi158.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-20T02:38:52.691 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-20T02:38:52.691 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-20T02:38:52.692 INFO:teuthology.orchestra.run.smithi158.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T02:38:52.692 INFO:teuthology.orchestra.run.smithi158.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T02:38:52.692 INFO:teuthology.orchestra.run.smithi158.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T02:38:52.692 INFO:teuthology.orchestra.run.smithi158.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-20T02:38:52.692 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-07-20T02:38:52.693 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2023-07-20T02:38:52.860 DEBUG:teuthology.parallel:result is None 2023-07-20T02:38:56.706 INFO:teuthology.orchestra.run.smithi007.stdout:Installed products updated. 2023-07-20T02:38:56.960 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:38:56.961 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-07-20T02:38:56.961 INFO:teuthology.orchestra.run.smithi007.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T02:38:56.961 INFO:teuthology.orchestra.run.smithi007.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T02:38:56.961 INFO:teuthology.orchestra.run.smithi007.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T02:38:56.961 INFO:teuthology.orchestra.run.smithi007.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-20T02:38:56.961 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-20T02:38:56.961 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-20T02:38:56.962 INFO:teuthology.orchestra.run.smithi007.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T02:38:56.962 INFO:teuthology.orchestra.run.smithi007.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T02:38:56.962 INFO:teuthology.orchestra.run.smithi007.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T02:38:56.962 INFO:teuthology.orchestra.run.smithi007.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-20T02:38:56.962 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:38:56.962 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-07-20T02:38:57.211 DEBUG:teuthology.parallel:result is None 2023-07-20T02:38:57.212 INFO:teuthology.run_tasks:Running task install... 2023-07-20T02:38:57.223 DEBUG:teuthology.task.install:project ceph 2023-07-20T02:38:57.224 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537'}} 2023-07-20T02:38:57.224 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537'} 2023-07-20T02:38:57.224 INFO:teuthology.task.install:Using flavor: default 2023-07-20T02:38:57.230 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-07-20T02:38:57.230 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-07-20T02:38:57.231 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'tag': None, 'wait_for_package': False} 2023-07-20T02:38:57.231 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T02:38:57.233 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'tag': None, 'wait_for_package': False} 2023-07-20T02:38:57.234 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T02:38:57.235 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'tag': None, 'wait_for_package': False} 2023-07-20T02:38:57.235 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T02:38:57.521 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/ 2023-07-20T02:38:57.522 INFO:teuthology.task.install.rpm:Package version is 18.0.0-5040.g1b0b2eb0 2023-07-20T02:38:57.559 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/ 2023-07-20T02:38:57.560 INFO:teuthology.task.install.rpm:Package version is 18.0.0-5040.g1b0b2eb0 2023-07-20T02:38:57.568 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/ 2023-07-20T02:38:57.568 INFO:teuthology.task.install.rpm:Package version is 18.0.0-5040.g1b0b2eb0 2023-07-20T02:38:57.806 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-20T02:38:57.807 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2023-07-20T02:38:57.807 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-20T02:38:57.811 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-20T02:38:57.812 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-07-20T02:38:57.812 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-20T02:38:57.815 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-20T02:38:57.815 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-07-20T02:38:57.815 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-20T02:38:57.833 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-20T02:38:57.834 DEBUG:teuthology.orchestra.run.smithi188:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1b0b2eb005a7c51d97a788b9900d511e1394b537/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-20T02:38:57.842 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-20T02:38:57.842 DEBUG:teuthology.orchestra.run.smithi158:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1b0b2eb005a7c51d97a788b9900d511e1394b537/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-20T02:38:57.885 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-20T02:38:57.886 DEBUG:teuthology.orchestra.run.smithi007:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1b0b2eb005a7c51d97a788b9900d511e1394b537/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-20T02:38:57.901 DEBUG:teuthology.orchestra.run.smithi188:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-20T02:38:57.910 DEBUG:teuthology.orchestra.run.smithi158:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-20T02:38:57.977 DEBUG:teuthology.orchestra.run.smithi188:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-20T02:38:57.986 DEBUG:teuthology.orchestra.run.smithi158:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-20T02:38:57.999 DEBUG:teuthology.orchestra.run.smithi007:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-20T02:38:58.042 INFO:teuthology.orchestra.run.smithi188.stdout:check_obsoletes = 1 2023-07-20T02:38:58.043 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum clean all 2023-07-20T02:38:58.051 INFO:teuthology.orchestra.run.smithi158.stdout:check_obsoletes = 1 2023-07-20T02:38:58.053 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum clean all 2023-07-20T02:38:58.098 DEBUG:teuthology.orchestra.run.smithi007:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-20T02:38:58.170 INFO:teuthology.orchestra.run.smithi007.stdout:check_obsoletes = 1 2023-07-20T02:38:58.173 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum clean all 2023-07-20T02:38:58.442 INFO:teuthology.orchestra.run.smithi158.stdout:Updating Subscription Management repositories. 2023-07-20T02:38:58.473 INFO:teuthology.orchestra.run.smithi188.stdout:Updating Subscription Management repositories. 2023-07-20T02:38:58.672 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-07-20T02:38:59.217 INFO:teuthology.orchestra.run.smithi158.stdout:44 files removed 2023-07-20T02:38:59.272 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-20T02:38:59.391 INFO:teuthology.orchestra.run.smithi007.stdout:44 files removed 2023-07-20T02:38:59.395 INFO:teuthology.orchestra.run.smithi188.stdout:44 files removed 2023-07-20T02:38:59.444 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-20T02:38:59.452 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-20T02:38:59.635 INFO:teuthology.orchestra.run.smithi158.stdout:Updating Subscription Management repositories. 2023-07-20T02:38:59.792 INFO:teuthology.orchestra.run.smithi188.stdout:Updating Subscription Management repositories. 2023-07-20T02:38:59.939 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-07-20T02:39:01.699 INFO:teuthology.orchestra.run.smithi158.stdout:ceph packages for x86_64 79 kB/s | 81 kB 00:01 2023-07-20T02:39:01.732 INFO:teuthology.orchestra.run.smithi188.stdout:ceph packages for x86_64 101 kB/s | 81 kB 00:00 2023-07-20T02:39:01.941 INFO:teuthology.orchestra.run.smithi007.stdout:ceph packages for x86_64 101 kB/s | 81 kB 00:00 2023-07-20T02:39:02.433 INFO:teuthology.orchestra.run.smithi188.stdout:ceph noarch packages 14 kB/s | 9.4 kB 00:00 2023-07-20T02:39:02.454 INFO:teuthology.orchestra.run.smithi158.stdout:ceph noarch packages 13 kB/s | 9.4 kB 00:00 2023-07-20T02:39:02.534 INFO:teuthology.orchestra.run.smithi007.stdout:ceph noarch packages 17 kB/s | 9.4 kB 00:00 2023-07-20T02:39:02.965 INFO:teuthology.orchestra.run.smithi158.stdout:ceph source packages 3.0 kB/s | 1.5 kB 00:00 2023-07-20T02:39:02.967 INFO:teuthology.orchestra.run.smithi188.stdout:ceph source packages 2.9 kB/s | 1.5 kB 00:00 2023-07-20T02:39:03.069 INFO:teuthology.orchestra.run.smithi007.stdout:ceph source packages 2.9 kB/s | 1.5 kB 00:00 2023-07-20T02:39:03.258 INFO:teuthology.orchestra.run.smithi158.stdout:Copr repo for python3-asyncssh owned by ceph 13 kB/s | 3.5 kB 00:00 2023-07-20T02:39:03.268 INFO:teuthology.orchestra.run.smithi188.stdout:Copr repo for python3-asyncssh owned by ceph 12 kB/s | 3.5 kB 00:00 2023-07-20T02:39:03.428 INFO:teuthology.orchestra.run.smithi007.stdout:Copr repo for python3-asyncssh owned by ceph 10 kB/s | 3.5 kB 00:00 2023-07-20T02:39:04.836 INFO:teuthology.orchestra.run.smithi158.stdout:Extra Packages for Enterprise Linux 10 MB/s | 16 MB 00:01 2023-07-20T02:39:05.260 INFO:teuthology.orchestra.run.smithi188.stdout:Extra Packages for Enterprise Linux 7.9 MB/s | 16 MB 00:01 2023-07-20T02:39:08.337 INFO:teuthology.orchestra.run.smithi158.stdout:lab-extras 333 kB/s | 24 kB 00:00 2023-07-20T02:39:08.727 INFO:teuthology.orchestra.run.smithi188.stdout:lab-extras 394 kB/s | 24 kB 00:00 2023-07-20T02:39:09.638 INFO:teuthology.orchestra.run.smithi158.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 43 MB/s | 53 MB 00:01 2023-07-20T02:39:09.978 INFO:teuthology.orchestra.run.smithi188.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 45 MB/s | 53 MB 00:01 2023-07-20T02:39:18.313 INFO:teuthology.orchestra.run.smithi188.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 44 MB/s | 47 MB 00:01 2023-07-20T02:39:18.548 INFO:teuthology.orchestra.run.smithi158.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 30 MB/s | 47 MB 00:01 2023-07-20T02:39:29.203 INFO:teuthology.orchestra.run.smithi188.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T02:39:29.203 INFO:teuthology.orchestra.run.smithi188.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T02:39:29.218 INFO:teuthology.orchestra.run.smithi188.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-20T02:39:29.219 INFO:teuthology.orchestra.run.smithi188.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-20T02:39:29.220 INFO:teuthology.orchestra.run.smithi188.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-20T02:39:29.221 INFO:teuthology.orchestra.run.smithi188.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-20T02:39:29.224 INFO:teuthology.orchestra.run.smithi188.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-20T02:39:29.227 INFO:teuthology.orchestra.run.smithi188.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-20T02:39:29.228 INFO:teuthology.orchestra.run.smithi188.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-20T02:39:29.361 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2023-07-20T02:39:29.373 INFO:teuthology.orchestra.run.smithi188.stdout:=========================================================================================================================== 2023-07-20T02:39:29.374 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2023-07-20T02:39:29.374 INFO:teuthology.orchestra.run.smithi188.stdout:=========================================================================================================================== 2023-07-20T02:39:29.374 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2023-07-20T02:39:29.374 INFO:teuthology.orchestra.run.smithi188.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-20T02:39:29.375 INFO:teuthology.orchestra.run.smithi188.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-20T02:39:29.375 INFO:teuthology.orchestra.run.smithi188.stdout: ceph x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.4 k 2023-07-20T02:39:29.375 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 5.4 M 2023-07-20T02:39:29.375 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 850 k 2023-07-20T02:39:29.375 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 167 k 2023-07-20T02:39:29.375 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 1.6 M 2023-07-20T02:39:29.375 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 129 k 2023-07-20T02:39:29.376 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-dashboard noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 1.7 M 2023-07-20T02:39:29.376 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 7.4 M 2023-07-20T02:39:29.376 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-rook noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 47 k 2023-07-20T02:39:29.376 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-radosgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 14 M 2023-07-20T02:39:29.376 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-test x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 48 M 2023-07-20T02:39:29.376 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-volume noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 258 k 2023-07-20T02:39:29.377 INFO:teuthology.orchestra.run.smithi188.stdout: cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 165 k 2023-07-20T02:39:29.377 INFO:teuthology.orchestra.run.smithi188.stdout: cephfs-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 260 k 2023-07-20T02:39:29.377 INFO:teuthology.orchestra.run.smithi188.stdout: cephfs-top noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 18 k 2023-07-20T02:39:29.377 INFO:teuthology.orchestra.run.smithi188.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-20T02:39:29.377 INFO:teuthology.orchestra.run.smithi188.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-20T02:39:29.378 INFO:teuthology.orchestra.run.smithi188.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-20T02:39:29.378 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 31 k 2023-07-20T02:39:29.378 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 771 k 2023-07-20T02:39:29.378 INFO:teuthology.orchestra.run.smithi188.stdout: librados-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T02:39:29.379 INFO:teuthology.orchestra.run.smithi188.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-20T02:39:29.379 INFO:teuthology.orchestra.run.smithi188.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-20T02:39:29.379 INFO:teuthology.orchestra.run.smithi188.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-20T02:39:29.379 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-20T02:39:29.379 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-20T02:39:29.379 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-20T02:39:29.379 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cephfs x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 197 k 2023-07-20T02:39:29.379 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rados x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 370 k 2023-07-20T02:39:29.380 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 363 k 2023-07-20T02:39:29.380 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 107 k 2023-07-20T02:39:29.380 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 94 k 2023-07-20T02:39:29.380 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.5 M 2023-07-20T02:39:29.380 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-nbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 179 k 2023-07-20T02:39:29.380 INFO:teuthology.orchestra.run.smithi188.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-20T02:39:29.380 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrading: 2023-07-20T02:39:29.381 INFO:teuthology.orchestra.run.smithi188.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-20T02:39:29.381 INFO:teuthology.orchestra.run.smithi188.stdout: librados2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.7 M 2023-07-20T02:39:29.381 INFO:teuthology.orchestra.run.smithi188.stdout: librbd1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.6 M 2023-07-20T02:39:29.381 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2023-07-20T02:39:29.382 INFO:teuthology.orchestra.run.smithi188.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-20T02:39:29.382 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 22 M 2023-07-20T02:39:29.382 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-grafana-dashboards noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 23 k 2023-07-20T02:39:29.382 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mds x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 2.3 M 2023-07-20T02:39:29.382 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 241 k 2023-07-20T02:39:29.383 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mon x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 4.6 M 2023-07-20T02:39:29.383 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-osd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 18 M 2023-07-20T02:39:29.383 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-prometheus-alerts noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 15 k 2023-07-20T02:39:29.383 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-selinux x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 24 k 2023-07-20T02:39:29.383 INFO:teuthology.orchestra.run.smithi188.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-20T02:39:29.383 INFO:teuthology.orchestra.run.smithi188.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-20T02:39:29.383 INFO:teuthology.orchestra.run.smithi188.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-20T02:39:29.383 INFO:teuthology.orchestra.run.smithi188.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-20T02:39:29.384 INFO:teuthology.orchestra.run.smithi188.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-20T02:39:29.384 INFO:teuthology.orchestra.run.smithi188.stdout: libcephsqlite x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T02:39:29.384 INFO:teuthology.orchestra.run.smithi188.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-20T02:39:29.384 INFO:teuthology.orchestra.run.smithi188.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-20T02:39:29.384 INFO:teuthology.orchestra.run.smithi188.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-20T02:39:29.385 INFO:teuthology.orchestra.run.smithi188.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-20T02:39:29.385 INFO:teuthology.orchestra.run.smithi188.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-20T02:39:29.385 INFO:teuthology.orchestra.run.smithi188.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-20T02:39:29.385 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 472 k 2023-07-20T02:39:29.385 INFO:teuthology.orchestra.run.smithi188.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-20T02:39:29.385 INFO:teuthology.orchestra.run.smithi188.stdout: librgw2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.8 M 2023-07-20T02:39:29.385 INFO:teuthology.orchestra.run.smithi188.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-20T02:39:29.386 INFO:teuthology.orchestra.run.smithi188.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-20T02:39:29.386 INFO:teuthology.orchestra.run.smithi188.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-20T02:39:29.386 INFO:teuthology.orchestra.run.smithi188.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-20T02:39:29.386 INFO:teuthology.orchestra.run.smithi188.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-20T02:39:29.386 INFO:teuthology.orchestra.run.smithi188.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-20T02:39:29.386 INFO:teuthology.orchestra.run.smithi188.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-20T02:39:29.386 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-20T02:39:29.387 INFO:teuthology.orchestra.run.smithi188.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-20T02:39:29.387 INFO:teuthology.orchestra.run.smithi188.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-20T02:39:29.387 INFO:teuthology.orchestra.run.smithi188.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-20T02:39:29.387 INFO:teuthology.orchestra.run.smithi188.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-20T02:39:29.387 INFO:teuthology.orchestra.run.smithi188.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-20T02:39:29.387 INFO:teuthology.orchestra.run.smithi188.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-20T02:39:29.388 INFO:teuthology.orchestra.run.smithi188.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-20T02:39:29.388 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-20T02:39:29.388 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-argparse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 45 k 2023-07-20T02:39:29.388 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 122 k 2023-07-20T02:39:29.388 INFO:teuthology.orchestra.run.smithi188.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-20T02:39:29.389 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-20T02:39:29.389 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-20T02:39:29.389 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-20T02:39:29.389 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-20T02:39:29.389 INFO:teuthology.orchestra.run.smithi188.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-20T02:39:29.390 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-20T02:39:29.390 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-20T02:39:29.390 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-20T02:39:29.390 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-20T02:39:29.390 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-20T02:39:29.390 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T02:39:29.390 INFO:teuthology.orchestra.run.smithi188.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-20T02:39:29.391 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-20T02:39:29.391 INFO:teuthology.orchestra.run.smithi188.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-20T02:39:29.391 INFO:teuthology.orchestra.run.smithi188.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-20T02:39:29.391 INFO:teuthology.orchestra.run.smithi188.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-20T02:39:29.391 INFO:teuthology.orchestra.run.smithi188.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-20T02:39:29.391 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-20T02:39:29.392 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-20T02:39:29.392 INFO:teuthology.orchestra.run.smithi188.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-20T02:39:29.392 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-20T02:39:29.392 INFO:teuthology.orchestra.run.smithi188.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-20T02:39:29.392 INFO:teuthology.orchestra.run.smithi188.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-20T02:39:29.392 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-20T02:39:29.392 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-20T02:39:29.393 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-20T02:39:29.393 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-20T02:39:29.393 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-20T02:39:29.393 INFO:teuthology.orchestra.run.smithi188.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-20T02:39:29.393 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T02:39:29.393 INFO:teuthology.orchestra.run.smithi188.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-20T02:39:29.393 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-20T02:39:29.394 INFO:teuthology.orchestra.run.smithi188.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-20T02:39:29.394 INFO:teuthology.orchestra.run.smithi188.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-20T02:39:29.394 INFO:teuthology.orchestra.run.smithi188.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-20T02:39:29.394 INFO:teuthology.orchestra.run.smithi188.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-20T02:39:29.394 INFO:teuthology.orchestra.run.smithi188.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-20T02:39:29.394 INFO:teuthology.orchestra.run.smithi188.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-20T02:39:29.394 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-20T02:39:29.395 INFO:teuthology.orchestra.run.smithi188.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-20T02:39:29.395 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-20T02:39:29.395 INFO:teuthology.orchestra.run.smithi188.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-20T02:39:29.395 INFO:teuthology.orchestra.run.smithi188.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-20T02:39:29.395 INFO:teuthology.orchestra.run.smithi188.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-20T02:39:29.396 INFO:teuthology.orchestra.run.smithi188.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-20T02:39:29.396 INFO:teuthology.orchestra.run.smithi188.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-20T02:39:29.396 INFO:teuthology.orchestra.run.smithi188.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-20T02:39:29.396 INFO:teuthology.orchestra.run.smithi188.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-20T02:39:29.396 INFO:teuthology.orchestra.run.smithi188.stdout:Enabling module streams: 2023-07-20T02:39:29.396 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql 10 2023-07-20T02:39:29.396 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-07-20T02:39:29.397 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2023-07-20T02:39:29.397 INFO:teuthology.orchestra.run.smithi188.stdout:=========================================================================================================================== 2023-07-20T02:39:29.397 INFO:teuthology.orchestra.run.smithi188.stdout:Install 128 Packages 2023-07-20T02:39:29.397 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrade 3 Packages 2023-07-20T02:39:29.397 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-07-20T02:39:29.398 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 206 M 2023-07-20T02:39:29.398 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2023-07-20T02:39:29.447 INFO:teuthology.orchestra.run.smithi158.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T02:39:29.448 INFO:teuthology.orchestra.run.smithi158.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T02:39:29.462 INFO:teuthology.orchestra.run.smithi158.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-20T02:39:29.463 INFO:teuthology.orchestra.run.smithi158.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-20T02:39:29.464 INFO:teuthology.orchestra.run.smithi158.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-20T02:39:29.465 INFO:teuthology.orchestra.run.smithi158.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-20T02:39:29.468 INFO:teuthology.orchestra.run.smithi158.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-20T02:39:29.471 INFO:teuthology.orchestra.run.smithi158.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-20T02:39:29.472 INFO:teuthology.orchestra.run.smithi158.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-20T02:39:29.611 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2023-07-20T02:39:29.623 INFO:teuthology.orchestra.run.smithi158.stdout:=========================================================================================================================== 2023-07-20T02:39:29.623 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repository Size 2023-07-20T02:39:29.623 INFO:teuthology.orchestra.run.smithi158.stdout:=========================================================================================================================== 2023-07-20T02:39:29.624 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2023-07-20T02:39:29.624 INFO:teuthology.orchestra.run.smithi158.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-20T02:39:29.624 INFO:teuthology.orchestra.run.smithi158.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-20T02:39:29.624 INFO:teuthology.orchestra.run.smithi158.stdout: ceph x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.4 k 2023-07-20T02:39:29.624 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-base x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 5.4 M 2023-07-20T02:39:29.624 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 850 k 2023-07-20T02:39:29.624 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 167 k 2023-07-20T02:39:29.625 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 1.6 M 2023-07-20T02:39:29.625 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 129 k 2023-07-20T02:39:29.625 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-dashboard noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 1.7 M 2023-07-20T02:39:29.625 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 7.4 M 2023-07-20T02:39:29.625 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-rook noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 47 k 2023-07-20T02:39:29.625 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-radosgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 14 M 2023-07-20T02:39:29.625 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-test x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 48 M 2023-07-20T02:39:29.626 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-volume noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 258 k 2023-07-20T02:39:29.626 INFO:teuthology.orchestra.run.smithi158.stdout: cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 165 k 2023-07-20T02:39:29.626 INFO:teuthology.orchestra.run.smithi158.stdout: cephfs-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 260 k 2023-07-20T02:39:29.626 INFO:teuthology.orchestra.run.smithi158.stdout: cephfs-top noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 18 k 2023-07-20T02:39:29.626 INFO:teuthology.orchestra.run.smithi158.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-20T02:39:29.626 INFO:teuthology.orchestra.run.smithi158.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-20T02:39:29.626 INFO:teuthology.orchestra.run.smithi158.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-20T02:39:29.627 INFO:teuthology.orchestra.run.smithi158.stdout: libcephfs-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 31 k 2023-07-20T02:39:29.627 INFO:teuthology.orchestra.run.smithi158.stdout: libcephfs2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 771 k 2023-07-20T02:39:29.627 INFO:teuthology.orchestra.run.smithi158.stdout: librados-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T02:39:29.627 INFO:teuthology.orchestra.run.smithi158.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-20T02:39:29.627 INFO:teuthology.orchestra.run.smithi158.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-20T02:39:29.627 INFO:teuthology.orchestra.run.smithi158.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-20T02:39:29.627 INFO:teuthology.orchestra.run.smithi158.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-20T02:39:29.627 INFO:teuthology.orchestra.run.smithi158.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-20T02:39:29.628 INFO:teuthology.orchestra.run.smithi158.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-20T02:39:29.628 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cephfs x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 197 k 2023-07-20T02:39:29.628 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rados x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 370 k 2023-07-20T02:39:29.628 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 363 k 2023-07-20T02:39:29.628 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 107 k 2023-07-20T02:39:29.628 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 94 k 2023-07-20T02:39:29.628 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.5 M 2023-07-20T02:39:29.629 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-nbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 179 k 2023-07-20T02:39:29.629 INFO:teuthology.orchestra.run.smithi158.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-20T02:39:29.629 INFO:teuthology.orchestra.run.smithi158.stdout:Upgrading: 2023-07-20T02:39:29.629 INFO:teuthology.orchestra.run.smithi158.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-20T02:39:29.629 INFO:teuthology.orchestra.run.smithi158.stdout: librados2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.7 M 2023-07-20T02:39:29.629 INFO:teuthology.orchestra.run.smithi158.stdout: librbd1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.6 M 2023-07-20T02:39:29.630 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2023-07-20T02:39:29.630 INFO:teuthology.orchestra.run.smithi158.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-20T02:39:29.630 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 22 M 2023-07-20T02:39:29.630 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-grafana-dashboards noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 23 k 2023-07-20T02:39:29.630 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mds x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 2.3 M 2023-07-20T02:39:29.630 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-modules-core noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 241 k 2023-07-20T02:39:29.630 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mon x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 4.6 M 2023-07-20T02:39:29.630 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-osd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 18 M 2023-07-20T02:39:29.631 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-prometheus-alerts noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 15 k 2023-07-20T02:39:29.631 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-selinux x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 24 k 2023-07-20T02:39:29.631 INFO:teuthology.orchestra.run.smithi158.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-20T02:39:29.631 INFO:teuthology.orchestra.run.smithi158.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-20T02:39:29.631 INFO:teuthology.orchestra.run.smithi158.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-20T02:39:29.631 INFO:teuthology.orchestra.run.smithi158.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-20T02:39:29.631 INFO:teuthology.orchestra.run.smithi158.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-20T02:39:29.631 INFO:teuthology.orchestra.run.smithi158.stdout: libcephsqlite x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T02:39:29.632 INFO:teuthology.orchestra.run.smithi158.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-20T02:39:29.632 INFO:teuthology.orchestra.run.smithi158.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-20T02:39:29.632 INFO:teuthology.orchestra.run.smithi158.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-20T02:39:29.632 INFO:teuthology.orchestra.run.smithi158.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-20T02:39:29.632 INFO:teuthology.orchestra.run.smithi158.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-20T02:39:29.632 INFO:teuthology.orchestra.run.smithi158.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-20T02:39:29.632 INFO:teuthology.orchestra.run.smithi158.stdout: libradosstriper1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 472 k 2023-07-20T02:39:29.633 INFO:teuthology.orchestra.run.smithi158.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-20T02:39:29.633 INFO:teuthology.orchestra.run.smithi158.stdout: librgw2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.8 M 2023-07-20T02:39:29.633 INFO:teuthology.orchestra.run.smithi158.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-20T02:39:29.633 INFO:teuthology.orchestra.run.smithi158.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-20T02:39:29.633 INFO:teuthology.orchestra.run.smithi158.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-20T02:39:29.633 INFO:teuthology.orchestra.run.smithi158.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-20T02:39:29.633 INFO:teuthology.orchestra.run.smithi158.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-20T02:39:29.634 INFO:teuthology.orchestra.run.smithi158.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-20T02:39:29.634 INFO:teuthology.orchestra.run.smithi158.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-20T02:39:29.634 INFO:teuthology.orchestra.run.smithi158.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-20T02:39:29.634 INFO:teuthology.orchestra.run.smithi158.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-20T02:39:29.634 INFO:teuthology.orchestra.run.smithi158.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-20T02:39:29.634 INFO:teuthology.orchestra.run.smithi158.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-20T02:39:29.634 INFO:teuthology.orchestra.run.smithi158.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-20T02:39:29.634 INFO:teuthology.orchestra.run.smithi158.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-20T02:39:29.635 INFO:teuthology.orchestra.run.smithi158.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-20T02:39:29.635 INFO:teuthology.orchestra.run.smithi158.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-20T02:39:29.635 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-20T02:39:29.635 INFO:teuthology.orchestra.run.smithi158.stdout: python3-ceph-argparse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 45 k 2023-07-20T02:39:29.635 INFO:teuthology.orchestra.run.smithi158.stdout: python3-ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 122 k 2023-07-20T02:39:29.636 INFO:teuthology.orchestra.run.smithi158.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-20T02:39:29.636 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-20T02:39:29.636 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-20T02:39:29.637 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-20T02:39:29.637 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-20T02:39:29.637 INFO:teuthology.orchestra.run.smithi158.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-20T02:39:29.637 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-20T02:39:29.638 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-20T02:39:29.638 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-20T02:39:29.638 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-20T02:39:29.639 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-20T02:39:29.639 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T02:39:29.639 INFO:teuthology.orchestra.run.smithi158.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-20T02:39:29.640 INFO:teuthology.orchestra.run.smithi158.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-20T02:39:29.640 INFO:teuthology.orchestra.run.smithi158.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-20T02:39:29.640 INFO:teuthology.orchestra.run.smithi158.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-20T02:39:29.640 INFO:teuthology.orchestra.run.smithi158.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-20T02:39:29.641 INFO:teuthology.orchestra.run.smithi158.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-20T02:39:29.641 INFO:teuthology.orchestra.run.smithi158.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-20T02:39:29.641 INFO:teuthology.orchestra.run.smithi158.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-20T02:39:29.641 INFO:teuthology.orchestra.run.smithi158.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-20T02:39:29.641 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-20T02:39:29.641 INFO:teuthology.orchestra.run.smithi158.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-20T02:39:29.642 INFO:teuthology.orchestra.run.smithi158.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-20T02:39:29.642 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-20T02:39:29.642 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-20T02:39:29.642 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-20T02:39:29.642 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-20T02:39:29.642 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-20T02:39:29.642 INFO:teuthology.orchestra.run.smithi158.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-20T02:39:29.643 INFO:teuthology.orchestra.run.smithi158.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T02:39:29.643 INFO:teuthology.orchestra.run.smithi158.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-20T02:39:29.643 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-20T02:39:29.643 INFO:teuthology.orchestra.run.smithi158.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-20T02:39:29.643 INFO:teuthology.orchestra.run.smithi158.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-20T02:39:29.644 INFO:teuthology.orchestra.run.smithi158.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-20T02:39:29.644 INFO:teuthology.orchestra.run.smithi158.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-20T02:39:29.644 INFO:teuthology.orchestra.run.smithi158.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-20T02:39:29.644 INFO:teuthology.orchestra.run.smithi158.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-20T02:39:29.644 INFO:teuthology.orchestra.run.smithi158.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-20T02:39:29.645 INFO:teuthology.orchestra.run.smithi158.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-20T02:39:29.645 INFO:teuthology.orchestra.run.smithi158.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-20T02:39:29.645 INFO:teuthology.orchestra.run.smithi158.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-20T02:39:29.645 INFO:teuthology.orchestra.run.smithi158.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-20T02:39:29.645 INFO:teuthology.orchestra.run.smithi158.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-20T02:39:29.645 INFO:teuthology.orchestra.run.smithi158.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-20T02:39:29.645 INFO:teuthology.orchestra.run.smithi158.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-20T02:39:29.646 INFO:teuthology.orchestra.run.smithi158.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-20T02:39:29.646 INFO:teuthology.orchestra.run.smithi158.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-20T02:39:29.646 INFO:teuthology.orchestra.run.smithi158.stdout:Enabling module streams: 2023-07-20T02:39:29.646 INFO:teuthology.orchestra.run.smithi158.stdout: postgresql 10 2023-07-20T02:39:29.646 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-07-20T02:39:29.647 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2023-07-20T02:39:29.647 INFO:teuthology.orchestra.run.smithi158.stdout:=========================================================================================================================== 2023-07-20T02:39:29.647 INFO:teuthology.orchestra.run.smithi158.stdout:Install 128 Packages 2023-07-20T02:39:29.647 INFO:teuthology.orchestra.run.smithi158.stdout:Upgrade 3 Packages 2023-07-20T02:39:29.647 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-07-20T02:39:29.648 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 206 M 2023-07-20T02:39:29.648 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2023-07-20T02:39:29.880 INFO:teuthology.orchestra.run.smithi188.stdout:(1/131): ceph-18.0.0-5040.g1b0b2eb0.el8.x86_64. 20 kB/s | 6.4 kB 00:00 2023-07-20T02:39:30.233 INFO:teuthology.orchestra.run.smithi158.stdout:(1/131): ceph-18.0.0-5040.g1b0b2eb0.el8.x86_64. 20 kB/s | 6.4 kB 00:00 2023-07-20T02:39:30.400 INFO:teuthology.orchestra.run.smithi188.stdout:(2/131): ceph-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 1.6 MB/s | 850 kB 00:00 2023-07-20T02:39:30.489 INFO:teuthology.orchestra.run.smithi188.stdout:(3/131): ceph-immutable-object-cache-18.0.0-504 1.8 MB/s | 167 kB 00:00 2023-07-20T02:39:30.733 INFO:teuthology.orchestra.run.smithi158.stdout:(2/131): ceph-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 1.7 MB/s | 850 kB 00:00 2023-07-20T02:39:30.842 INFO:teuthology.orchestra.run.smithi158.stdout:(3/131): ceph-immutable-object-cache-18.0.0-504 1.5 MB/s | 167 kB 00:00 2023-07-20T02:39:31.156 INFO:teuthology.orchestra.run.smithi188.stdout:(4/131): ceph-mds-18.0.0-5040.g1b0b2eb0.el8.x86 3.4 MB/s | 2.3 MB 00:00 2023-07-20T02:39:31.200 INFO:teuthology.orchestra.run.smithi158.stdout:(4/131): ceph-base-18.0.0-5040.g1b0b2eb0.el8.x8 4.2 MB/s | 5.4 MB 00:01 2023-07-20T02:39:31.434 INFO:teuthology.orchestra.run.smithi158.stdout:(5/131): ceph-mds-18.0.0-5040.g1b0b2eb0.el8.x86 3.9 MB/s | 2.3 MB 00:00 2023-07-20T02:39:31.576 INFO:teuthology.orchestra.run.smithi158.stdout:(6/131): ceph-mgr-18.0.0-5040.g1b0b2eb0.el8.x86 4.1 MB/s | 1.6 MB 00:00 2023-07-20T02:39:31.639 INFO:teuthology.orchestra.run.smithi188.stdout:(5/131): ceph-mgr-18.0.0-5040.g1b0b2eb0.el8.x86 3.2 MB/s | 1.6 MB 00:00 2023-07-20T02:39:32.676 INFO:teuthology.orchestra.run.smithi158.stdout:(7/131): ceph-mon-18.0.0-5040.g1b0b2eb0.el8.x86 3.7 MB/s | 4.6 MB 00:01 2023-07-20T02:39:32.864 INFO:teuthology.orchestra.run.smithi188.stdout:(6/131): ceph-mon-18.0.0-5040.g1b0b2eb0.el8.x86 3.8 MB/s | 4.6 MB 00:01 2023-07-20T02:39:33.973 INFO:teuthology.orchestra.run.smithi188.stdout:(7/131): ceph-base-18.0.0-5040.g1b0b2eb0.el8.x8 1.2 MB/s | 5.4 MB 00:04 2023-07-20T02:39:34.577 INFO:teuthology.orchestra.run.smithi158.stdout:(8/131): ceph-common-18.0.0-5040.g1b0b2eb0.el8. 4.8 MB/s | 22 MB 00:04 2023-07-20T02:39:34.643 INFO:teuthology.orchestra.run.smithi158.stdout:(9/131): ceph-selinux-18.0.0-5040.g1b0b2eb0.el8 366 kB/s | 24 kB 00:00 2023-07-20T02:39:35.618 INFO:teuthology.orchestra.run.smithi158.stdout:(10/131): ceph-radosgw-18.0.0-5040.g1b0b2eb0.el 4.6 MB/s | 14 MB 00:02 2023-07-20T02:39:35.927 INFO:teuthology.orchestra.run.smithi158.stdout:(11/131): ceph-osd-18.0.0-5040.g1b0b2eb0.el8.x8 4.1 MB/s | 18 MB 00:04 2023-07-20T02:39:35.952 INFO:teuthology.orchestra.run.smithi158.stdout:(12/131): cephfs-mirror-18.0.0-5040.g1b0b2eb0.e 780 kB/s | 260 kB 00:00 2023-07-20T02:39:36.070 INFO:teuthology.orchestra.run.smithi158.stdout:(13/131): libcephfs-devel-18.0.0-5040.g1b0b2eb0 219 kB/s | 31 kB 00:00 2023-07-20T02:39:36.170 INFO:teuthology.orchestra.run.smithi158.stdout:(14/131): libcephsqlite-18.0.0-5040.g1b0b2eb0.e 1.4 MB/s | 139 kB 00:00 2023-07-20T02:39:36.220 INFO:teuthology.orchestra.run.smithi158.stdout:(15/131): libcephfs2-18.0.0-5040.g1b0b2eb0.el8. 2.8 MB/s | 771 kB 00:00 2023-07-20T02:39:36.237 INFO:teuthology.orchestra.run.smithi158.stdout:(16/131): librados-devel-18.0.0-5040.g1b0b2eb0. 2.0 MB/s | 139 kB 00:00 2023-07-20T02:39:36.379 INFO:teuthology.orchestra.run.smithi158.stdout:(17/131): libradosstriper1-18.0.0-5040.g1b0b2eb 2.9 MB/s | 472 kB 00:00 2023-07-20T02:39:36.454 INFO:teuthology.orchestra.run.smithi158.stdout:(18/131): python3-ceph-argparse-18.0.0-5040.g1b 598 kB/s | 45 kB 00:00 2023-07-20T02:39:36.571 INFO:teuthology.orchestra.run.smithi158.stdout:(19/131): python3-ceph-common-18.0.0-5040.g1b0b 1.0 MB/s | 122 kB 00:00 2023-07-20T02:39:36.646 INFO:teuthology.orchestra.run.smithi158.stdout:(20/131): python3-cephfs-18.0.0-5040.g1b0b2eb0. 2.6 MB/s | 197 kB 00:00 2023-07-20T02:39:36.822 INFO:teuthology.orchestra.run.smithi158.stdout:(21/131): python3-rados-18.0.0-5040.g1b0b2eb0.e 2.1 MB/s | 370 kB 00:00 2023-07-20T02:39:36.972 INFO:teuthology.orchestra.run.smithi158.stdout:(22/131): python3-rbd-18.0.0-5040.g1b0b2eb0.el8 2.4 MB/s | 363 kB 00:00 2023-07-20T02:39:37.064 INFO:teuthology.orchestra.run.smithi158.stdout:(23/131): python3-rgw-18.0.0-5040.g1b0b2eb0.el8 1.1 MB/s | 107 kB 00:00 2023-07-20T02:39:37.131 INFO:teuthology.orchestra.run.smithi158.stdout:(24/131): rbd-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 1.4 MB/s | 94 kB 00:00 2023-07-20T02:39:37.865 INFO:teuthology.orchestra.run.smithi158.stdout:(25/131): rbd-mirror-18.0.0-5040.g1b0b2eb0.el8. 4.8 MB/s | 3.5 MB 00:00 2023-07-20T02:39:37.948 INFO:teuthology.orchestra.run.smithi188.stdout:(8/131): ceph-osd-18.0.0-5040.g1b0b2eb0.el8.x86 3.5 MB/s | 18 MB 00:05 2023-07-20T02:39:38.039 INFO:teuthology.orchestra.run.smithi188.stdout:(9/131): ceph-selinux-18.0.0-5040.g1b0b2eb0.el8 266 kB/s | 24 kB 00:00 2023-07-20T02:39:38.046 INFO:teuthology.orchestra.run.smithi158.stdout:(26/131): librgw2-18.0.0-5040.g1b0b2eb0.el8.x86 3.7 MB/s | 6.8 MB 00:01 2023-07-20T02:39:38.082 INFO:teuthology.orchestra.run.smithi158.stdout:(27/131): rbd-nbd-18.0.0-5040.g1b0b2eb0.el8.x86 824 kB/s | 179 kB 00:00 2023-07-20T02:39:38.149 INFO:teuthology.orchestra.run.smithi158.stdout:(28/131): ceph-grafana-dashboards-18.0.0-5040.g 225 kB/s | 23 kB 00:00 2023-07-20T02:39:38.183 INFO:teuthology.orchestra.run.smithi158.stdout:(29/131): ceph-mgr-cephadm-18.0.0-5040.g1b0b2eb 1.3 MB/s | 129 kB 00:00 2023-07-20T02:39:38.817 INFO:teuthology.orchestra.run.smithi158.stdout:(30/131): ceph-mgr-dashboard-18.0.0-5040.g1b0b2 2.6 MB/s | 1.7 MB 00:00 2023-07-20T02:39:38.967 INFO:teuthology.orchestra.run.smithi158.stdout:(31/131): ceph-mgr-modules-core-18.0.0-5040.g1b 1.6 MB/s | 241 kB 00:00 2023-07-20T02:39:39.050 INFO:teuthology.orchestra.run.smithi158.stdout:(32/131): ceph-mgr-rook-18.0.0-5040.g1b0b2eb0.e 569 kB/s | 47 kB 00:00 2023-07-20T02:39:39.151 INFO:teuthology.orchestra.run.smithi158.stdout:(33/131): ceph-prometheus-alerts-18.0.0-5040.g1 145 kB/s | 15 kB 00:00 2023-07-20T02:39:39.318 INFO:teuthology.orchestra.run.smithi158.stdout:(34/131): ceph-volume-18.0.0-5040.g1b0b2eb0.el8 1.5 MB/s | 258 kB 00:00 2023-07-20T02:39:39.427 INFO:teuthology.orchestra.run.smithi158.stdout:(35/131): cephadm-18.0.0-5040.g1b0b2eb0.el8.noa 1.5 MB/s | 165 kB 00:00 2023-07-20T02:39:39.518 INFO:teuthology.orchestra.run.smithi158.stdout:(36/131): cephfs-top-18.0.0-5040.g1b0b2eb0.el8. 196 kB/s | 18 kB 00:00 2023-07-20T02:39:39.811 INFO:teuthology.orchestra.run.smithi158.stdout:(37/131): ceph-mgr-diskprediction-local-18.0.0- 4.6 MB/s | 7.4 MB 00:01 2023-07-20T02:39:40.202 INFO:teuthology.orchestra.run.smithi158.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 150 kB/s | 59 kB 00:00 2023-07-20T02:39:40.227 INFO:teuthology.orchestra.run.smithi158.stdout:(39/131): gperftools-libs-2.7-9.el8.x86_64.rpm 432 kB/s | 306 kB 00:00 2023-07-20T02:39:40.303 INFO:teuthology.orchestra.run.smithi158.stdout:(40/131): libunwind-1.3.1-3.el8.x86_64.rpm 748 kB/s | 75 kB 00:00 2023-07-20T02:39:40.811 INFO:teuthology.orchestra.run.smithi158.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 86 kB/s | 44 kB 00:00 2023-07-20T02:39:40.887 INFO:teuthology.orchestra.run.smithi158.stdout:(42/131): python3-asyncssh-2.7.0-2.el8.noarch.r 722 kB/s | 476 kB 00:00 2023-07-20T02:39:40.995 INFO:teuthology.orchestra.run.smithi158.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 305 kB/s | 33 kB 00:00 2023-07-20T02:39:41.043 INFO:teuthology.orchestra.run.smithi188.stdout:(10/131): ceph-radosgw-18.0.0-5040.g1b0b2eb0.el 1.9 MB/s | 14 MB 00:07 2023-07-20T02:39:41.129 INFO:teuthology.orchestra.run.smithi158.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 122 kB/s | 16 kB 00:00 2023-07-20T02:39:41.206 INFO:teuthology.orchestra.run.smithi188.stdout:(11/131): cephfs-mirror-18.0.0-5040.g1b0b2eb0.e 1.6 MB/s | 260 kB 00:00 2023-07-20T02:39:41.238 INFO:teuthology.orchestra.run.smithi158.stdout:(45/131): python3-beautifulsoup4-4.6.3-2.el8.1. 434 kB/s | 185 kB 00:00 2023-07-20T02:39:41.315 INFO:teuthology.orchestra.run.smithi188.stdout:(12/131): libcephfs-devel-18.0.0-5040.g1b0b2eb0 286 kB/s | 31 kB 00:00 2023-07-20T02:39:41.463 INFO:teuthology.orchestra.run.smithi158.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 517 kB/s | 173 kB 00:00 2023-07-20T02:39:41.647 INFO:teuthology.orchestra.run.smithi158.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 443 kB/s | 81 kB 00:00 2023-07-20T02:39:41.739 INFO:teuthology.orchestra.run.smithi158.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 121 kB/s | 11 kB 00:00 2023-07-20T02:39:41.789 INFO:teuthology.orchestra.run.smithi158.stdout:(49/131): python3-cherrypy-18.4.0-1.el8.noarch. 698 kB/s | 384 kB 00:00 2023-07-20T02:39:41.815 INFO:teuthology.orchestra.run.smithi158.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 233 kB/s | 18 kB 00:00 2023-07-20T02:39:41.865 INFO:teuthology.orchestra.run.smithi188.stdout:(13/131): libcephfs2-18.0.0-5040.g1b0b2eb0.el8. 1.4 MB/s | 771 kB 00:00 2023-07-20T02:39:41.906 INFO:teuthology.orchestra.run.smithi158.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 532 kB/s | 49 kB 00:00 2023-07-20T02:39:41.999 INFO:teuthology.orchestra.run.smithi188.stdout:(14/131): libcephsqlite-18.0.0-5040.g1b0b2eb0.e 1.0 MB/s | 139 kB 00:00 2023-07-20T02:39:42.049 INFO:teuthology.orchestra.run.smithi158.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 414 kB/s | 59 kB 00:00 2023-07-20T02:39:42.124 INFO:teuthology.orchestra.run.smithi188.stdout:(15/131): librados-devel-18.0.0-5040.g1b0b2eb0. 1.1 MB/s | 139 kB 00:00 2023-07-20T02:39:42.199 INFO:teuthology.orchestra.run.smithi158.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 402 kB/s | 60 kB 00:00 2023-07-20T02:39:42.391 INFO:teuthology.orchestra.run.smithi188.stdout:(16/131): libradosstriper1-18.0.0-5040.g1b0b2eb 1.7 MB/s | 472 kB 00:00 2023-07-20T02:39:42.566 INFO:teuthology.orchestra.run.smithi158.stdout:(54/131): python3-pecan-1.3.2-9.el8.noarch.rpm 772 kB/s | 283 kB 00:00 2023-07-20T02:39:42.658 INFO:teuthology.orchestra.run.smithi158.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 170 kB/s | 16 kB 00:00 2023-07-20T02:39:42.758 INFO:teuthology.orchestra.run.smithi158.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 337 kB/s | 34 kB 00:00 2023-07-20T02:39:43.175 INFO:teuthology.orchestra.run.smithi158.stdout:(57/131): python3-routes-2.4.1-12.el8.noarch.rp 471 kB/s | 196 kB 00:00 2023-07-20T02:39:43.717 INFO:teuthology.orchestra.run.smithi158.stdout:(58/131): python3-rsa-4.8-1.el8.noarch.rpm 113 kB/s | 61 kB 00:00 2023-07-20T02:39:43.792 INFO:teuthology.orchestra.run.smithi158.stdout:(59/131): python3-simplegeneric-0.8.1-17.el8.no 250 kB/s | 19 kB 00:00 2023-07-20T02:39:43.893 INFO:teuthology.orchestra.run.smithi158.stdout:(60/131): python3-singledispatch-3.4.0.3-18.el8 239 kB/s | 24 kB 00:00 2023-07-20T02:39:43.960 INFO:teuthology.orchestra.run.smithi158.stdout:(61/131): python3-tempora-1.14.1-5.el8.noarch.r 439 kB/s | 29 kB 00:00 2023-07-20T02:39:44.044 INFO:teuthology.orchestra.run.smithi158.stdout:(62/131): python3-trustme-0.6.0-4.el8.noarch.rp 330 kB/s | 27 kB 00:00 2023-07-20T02:39:44.885 INFO:teuthology.orchestra.run.smithi158.stdout:(63/131): python3-waitress-1.4.3-1.el8.noarch.r 303 kB/s | 255 kB 00:00 2023-07-20T02:39:45.652 INFO:teuthology.orchestra.run.smithi158.stdout:(64/131): python3-kubernetes-11.0.0-6.el8.noarc 309 kB/s | 1.2 MB 00:03 2023-07-20T02:39:45.703 INFO:teuthology.orchestra.run.smithi158.stdout:(65/131): python3-webob-1.8.5-1.el8.1.noarch.rp 307 kB/s | 251 kB 00:00 2023-07-20T02:39:45.970 INFO:teuthology.orchestra.run.smithi158.stdout:(66/131): python3-webtest-2.0.33-1.el8.noarch.r 320 kB/s | 85 kB 00:00 2023-07-20T02:39:46.020 INFO:teuthology.orchestra.run.smithi158.stdout:(67/131): python3-websocket-client-0.56.0-5.el8 167 kB/s | 61 kB 00:00 2023-07-20T02:39:46.045 INFO:teuthology.orchestra.run.smithi158.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 305 kB/s | 23 kB 00:00 2023-07-20T02:39:46.071 INFO:teuthology.orchestra.run.smithi158.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-07-20T02:39:46.213 INFO:teuthology.orchestra.run.smithi158.stdout:(70/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 330 kB/s | 47 kB 00:00 2023-07-20T02:39:46.380 INFO:teuthology.orchestra.run.smithi158.stdout:(71/131): libselinux-devel-2.9-5.el8.x86_64.rpm 1.2 MB/s | 200 kB 00:00 2023-07-20T02:39:46.466 INFO:teuthology.orchestra.run.smithi188.stdout:(17/131): ceph-common-18.0.0-5040.g1b0b2eb0.el8 1.3 MB/s | 22 MB 00:16 2023-07-20T02:39:46.472 INFO:teuthology.orchestra.run.smithi158.stdout:(72/131): python3-jwt-1.6.1-2.el8.noarch.rpm 464 kB/s | 43 kB 00:00 2023-07-20T02:39:46.608 INFO:teuthology.orchestra.run.smithi188.stdout:(18/131): librgw2-18.0.0-5040.g1b0b2eb0.el8.x86 1.6 MB/s | 6.8 MB 00:04 2023-07-20T02:39:46.633 INFO:teuthology.orchestra.run.smithi188.stdout:(19/131): python3-ceph-argparse-18.0.0-5040.g1b 268 kB/s | 45 kB 00:00 2023-07-20T02:39:46.638 INFO:teuthology.orchestra.run.smithi158.stdout:(73/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.3 MB/s | 220 kB 00:00 2023-07-20T02:39:46.742 INFO:teuthology.orchestra.run.smithi188.stdout:(20/131): python3-ceph-common-18.0.0-5040.g1b0b 919 kB/s | 122 kB 00:00 2023-07-20T02:39:46.789 INFO:teuthology.orchestra.run.smithi158.stdout:(74/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 320 kB/s | 48 kB 00:00 2023-07-20T02:39:46.834 INFO:teuthology.orchestra.run.smithi188.stdout:(21/131): python3-cephfs-18.0.0-5040.g1b0b2eb0. 983 kB/s | 197 kB 00:00 2023-07-20T02:39:46.906 INFO:teuthology.orchestra.run.smithi158.stdout:(75/131): libsepol-devel-2.9-3.el8.x86_64.rpm 746 kB/s | 87 kB 00:00 2023-07-20T02:39:47.025 INFO:teuthology.orchestra.run.smithi188.stdout:(22/131): python3-rados-18.0.0-5040.g1b0b2eb0.e 1.3 MB/s | 370 kB 00:00 2023-07-20T02:39:47.031 INFO:teuthology.orchestra.run.smithi158.stdout:(76/131): python3-requests-oauthlib-1.0.0-1.el8 342 kB/s | 43 kB 00:00 2023-07-20T02:39:47.101 INFO:teuthology.orchestra.run.smithi188.stdout:(23/131): python3-rgw-18.0.0-5040.g1b0b2eb0.el8 1.4 MB/s | 107 kB 00:00 2023-07-20T02:39:47.143 INFO:teuthology.orchestra.run.smithi188.stdout:(24/131): python3-rbd-18.0.0-5040.g1b0b2eb0.el8 1.1 MB/s | 363 kB 00:00 2023-07-20T02:39:47.168 INFO:teuthology.orchestra.run.smithi188.stdout:(25/131): rbd-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 1.4 MB/s | 94 kB 00:00 2023-07-20T02:39:47.182 INFO:teuthology.orchestra.run.smithi158.stdout:(77/131): python3-pycparser-2.14-14.el8.noarch. 728 kB/s | 109 kB 00:00 2023-07-20T02:39:47.352 INFO:teuthology.orchestra.run.smithi188.stdout:(26/131): rbd-nbd-18.0.0-5040.g1b0b2eb0.el8.x86 974 kB/s | 179 kB 00:00 2023-07-20T02:39:47.380 INFO:teuthology.orchestra.run.smithi158.stdout:(78/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.1 MB/s | 229 kB 00:00 2023-07-20T02:39:47.427 INFO:teuthology.orchestra.run.smithi188.stdout:(27/131): ceph-grafana-dashboards-18.0.0-5040.g 308 kB/s | 23 kB 00:00 2023-07-20T02:39:47.457 INFO:teuthology.orchestra.run.smithi158.stdout:(79/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 7.1 MB/s | 560 kB 00:00 2023-07-20T02:39:47.544 INFO:teuthology.orchestra.run.smithi188.stdout:(28/131): ceph-mgr-cephadm-18.0.0-5040.g1b0b2eb 1.1 MB/s | 129 kB 00:00 2023-07-20T02:39:47.599 INFO:teuthology.orchestra.run.smithi158.stdout:(80/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 132 kB/s | 19 kB 00:00 2023-07-20T02:39:47.708 INFO:teuthology.orchestra.run.smithi158.stdout:(81/131): python3-oauthlib-2.1.0-1.el8.noarch.r 1.4 MB/s | 155 kB 00:00 2023-07-20T02:39:47.983 INFO:teuthology.orchestra.run.smithi158.stdout:(82/131): python3-cryptography-3.2.1-5.el8.x86_ 2.0 MB/s | 559 kB 00:00 2023-07-20T02:39:48.092 INFO:teuthology.orchestra.run.smithi158.stdout:(83/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 165 kB/s | 18 kB 00:00 2023-07-20T02:39:48.261 INFO:teuthology.orchestra.run.smithi188.stdout:(29/131): ceph-mgr-dashboard-18.0.0-5040.g1b0b2 2.4 MB/s | 1.7 MB 00:00 2023-07-20T02:39:48.301 INFO:teuthology.orchestra.run.smithi158.stdout:(84/131): pcre2-devel-10.32-2.el8.x86_64.rpm 2.8 MB/s | 605 kB 00:00 2023-07-20T02:39:48.451 INFO:teuthology.orchestra.run.smithi158.stdout:(85/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.5 MB/s | 238 kB 00:00 2023-07-20T02:39:48.703 INFO:teuthology.orchestra.run.smithi158.stdout:(86/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 9.2 MB/s | 2.3 MB 00:00 2023-07-20T02:39:48.852 INFO:teuthology.orchestra.run.smithi158.stdout:(87/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 489 kB/s | 72 kB 00:00 2023-07-20T02:39:48.977 INFO:teuthology.orchestra.run.smithi158.stdout:(88/131): fuse-2.9.7-15.el8.x86_64.rpm 662 kB/s | 83 kB 00:00 2023-07-20T02:39:49.152 INFO:teuthology.orchestra.run.smithi158.stdout:(89/131): elfutils-libelf-devel-0.186-1.el8.x86 343 kB/s | 60 kB 00:00 2023-07-20T02:39:49.153 INFO:teuthology.orchestra.run.smithi188.stdout:(30/131): rbd-mirror-18.0.0-5040.g1b0b2eb0.el8. 1.8 MB/s | 3.5 MB 00:02 2023-07-20T02:39:49.253 INFO:teuthology.orchestra.run.smithi158.stdout:(90/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 981 kB/s | 98 kB 00:00 2023-07-20T02:39:49.411 INFO:teuthology.orchestra.run.smithi188.stdout:(31/131): ceph-mgr-modules-core-18.0.0-5040.g1b 934 kB/s | 241 kB 00:00 2023-07-20T02:39:49.537 INFO:teuthology.orchestra.run.smithi188.stdout:(32/131): ceph-mgr-rook-18.0.0-5040.g1b0b2eb0.e 380 kB/s | 47 kB 00:00 2023-07-20T02:39:49.629 INFO:teuthology.orchestra.run.smithi188.stdout:(33/131): ceph-prometheus-alerts-18.0.0-5040.g1 158 kB/s | 15 kB 00:00 2023-07-20T02:39:49.854 INFO:teuthology.orchestra.run.smithi188.stdout:(34/131): ceph-volume-18.0.0-5040.g1b0b2eb0.el8 1.1 MB/s | 258 kB 00:00 2023-07-20T02:39:49.886 INFO:teuthology.orchestra.run.smithi158.stdout:(91/131): ceph-test-18.0.0-5040.g1b0b2eb0.el8.x 3.1 MB/s | 48 MB 00:15 2023-07-20T02:39:49.912 INFO:teuthology.orchestra.run.smithi158.stdout:(92/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 87 kB/s | 58 kB 00:00 2023-07-20T02:39:50.021 INFO:teuthology.orchestra.run.smithi188.stdout:(35/131): cephadm-18.0.0-5040.g1b0b2eb0.el8.noa 991 kB/s | 165 kB 00:00 2023-07-20T02:39:50.113 INFO:teuthology.orchestra.run.smithi188.stdout:(36/131): cephfs-top-18.0.0-5040.g1b0b2eb0.el8. 196 kB/s | 18 kB 00:00 2023-07-20T02:39:50.121 INFO:teuthology.orchestra.run.smithi158.stdout:(93/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 3.1 MB/s | 644 kB 00:00 2023-07-20T02:39:50.163 INFO:teuthology.orchestra.run.smithi158.stdout:(94/131): libcom_err-devel-1.45.6-4.el8.x86_64. 140 kB/s | 39 kB 00:00 2023-07-20T02:39:50.310 INFO:teuthology.orchestra.run.smithi158.stdout:(95/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 905 kB/s | 171 kB 00:00 2023-07-20T02:39:50.346 INFO:teuthology.orchestra.run.smithi158.stdout:(96/131): NetworkManager-initscripts-updown-1.3 750 kB/s | 138 kB 00:00 2023-07-20T02:39:50.472 INFO:teuthology.orchestra.run.smithi158.stdout:(97/131): python3-pyasn1-modules-0.3.7-6.el8.no 881 kB/s | 110 kB 00:00 2023-07-20T02:39:50.607 INFO:teuthology.orchestra.run.smithi158.stdout:(98/131): python3-pytz-2017.2-9.el8.noarch.rpm 406 kB/s | 54 kB 00:00 2023-07-20T02:39:50.824 INFO:teuthology.orchestra.run.smithi158.stdout:(99/131): python3-scipy-1.0.0-21.module+el8.5.0 27 MB/s | 14 MB 00:00 2023-07-20T02:39:50.864 INFO:teuthology.orchestra.run.smithi158.stdout:(100/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.0 MB/s | 259 kB 00:00 2023-07-20T02:39:50.921 INFO:teuthology.orchestra.run.smithi188.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 379 kB/s | 306 kB 00:00 2023-07-20T02:39:50.923 INFO:teuthology.orchestra.run.smithi158.stdout:(101/131): socat-1.7.4.1-1.el8.x86_64.rpm 3.2 MB/s | 323 kB 00:00 2023-07-20T02:39:51.085 INFO:teuthology.orchestra.run.smithi158.stdout:(102/131): python3-markupsafe-0.23-19.el8.x86_6 176 kB/s | 39 kB 00:00 2023-07-20T02:39:51.132 INFO:teuthology.orchestra.run.smithi158.stdout:(103/131): bison-3.0.4-10.el8.x86_64.rpm 3.2 MB/s | 688 kB 00:00 2023-07-20T02:39:51.172 INFO:teuthology.orchestra.run.smithi188.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 235 kB/s | 59 kB 00:00 2023-07-20T02:39:51.241 INFO:teuthology.orchestra.run.smithi158.stdout:(104/131): python3-jinja2-2.10.1-3.el8.noarch.r 3.4 MB/s | 538 kB 00:00 2023-07-20T02:39:51.291 INFO:teuthology.orchestra.run.smithi158.stdout:(105/131): python3-mako-1.0.6-13.el8.noarch.rpm 989 kB/s | 157 kB 00:00 2023-07-20T02:39:51.297 INFO:teuthology.orchestra.run.smithi188.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 600 kB/s | 75 kB 00:00 2023-07-20T02:39:51.358 INFO:teuthology.orchestra.run.smithi158.stdout:(106/131): oniguruma-6.8.2-2.el8.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2023-07-20T02:39:51.417 INFO:teuthology.orchestra.run.smithi158.stdout:(107/131): automake-1.16.1-7.el8.noarch.rpm 5.6 MB/s | 713 kB 00:00 2023-07-20T02:39:51.556 INFO:teuthology.orchestra.run.smithi188.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 1.8 MB/s | 476 kB 00:00 2023-07-20T02:39:51.659 INFO:teuthology.orchestra.run.smithi158.stdout:(108/131): python3-babel-2.5.1-7.el8.noarch.rpm 16 MB/s | 4.8 MB 00:00 2023-07-20T02:39:51.673 INFO:teuthology.orchestra.run.smithi188.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 374 kB/s | 44 kB 00:00 2023-07-20T02:39:51.675 INFO:teuthology.orchestra.run.smithi158.stdout:(109/131): python3-jsonpatch-1.21-2.el8.noarch. 103 kB/s | 27 kB 00:00 2023-07-20T02:39:51.756 INFO:teuthology.orchestra.run.smithi188.stdout:(42/131): ceph-mgr-diskprediction-local-18.0.0- 2.1 MB/s | 7.4 MB 00:03 2023-07-20T02:39:51.823 INFO:teuthology.orchestra.run.smithi188.stdout:(43/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.2 MB/s | 185 kB 00:00 2023-07-20T02:39:51.901 INFO:teuthology.orchestra.run.smithi158.stdout:(110/131): python3-prettytable-0.7.2-14.el8.noa 195 kB/s | 44 kB 00:00 2023-07-20T02:39:51.926 INFO:teuthology.orchestra.run.smithi158.stdout:(111/131): flex-2.6.1-9.el8.x86_64.rpm 1.2 MB/s | 320 kB 00:00 2023-07-20T02:39:51.965 INFO:teuthology.orchestra.run.smithi188.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 115 kB/s | 16 kB 00:00 2023-07-20T02:39:52.091 INFO:teuthology.orchestra.run.smithi188.stdout:(45/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.3 MB/s | 173 kB 00:00 2023-07-20T02:39:52.185 INFO:teuthology.orchestra.run.smithi158.stdout:(112/131): python3-numpy-1.14.3-10.el8.x86_64.r 13 MB/s | 3.7 MB 00:00 2023-07-20T02:39:52.219 INFO:teuthology.orchestra.run.smithi158.stdout:(113/131): python3-werkzeug-0.12.2-4.el8.noarch 1.5 MB/s | 457 kB 00:00 2023-07-20T02:39:52.274 INFO:teuthology.orchestra.run.smithi188.stdout:(46/131): python3-cherrypy-18.4.0-1.el8.noarch. 2.1 MB/s | 384 kB 00:00 2023-07-20T02:39:52.300 INFO:teuthology.orchestra.run.smithi188.stdout:(47/131): python3-cachetools-3.1.1-4.el8.noarch 61 kB/s | 33 kB 00:00 2023-07-20T02:39:52.310 INFO:teuthology.orchestra.run.smithi158.stdout:(114/131): python3-numpy-f2py-1.14.3-10.el8.x86 1.8 MB/s | 225 kB 00:00 2023-07-20T02:39:52.400 INFO:teuthology.orchestra.run.smithi188.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 648 kB/s | 81 kB 00:00 2023-07-20T02:39:52.411 INFO:teuthology.orchestra.run.smithi158.stdout:(115/131): uuid-1.6.2-43.el8.x86_64.rpm 333 kB/s | 64 kB 00:00 2023-07-20T02:39:52.425 INFO:teuthology.orchestra.run.smithi188.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 88 kB/s | 11 kB 00:00 2023-07-20T02:39:52.444 INFO:teuthology.orchestra.run.smithi158.stdout:(116/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 945 kB/s | 126 kB 00:00 2023-07-20T02:39:52.509 INFO:teuthology.orchestra.run.smithi188.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 161 kB/s | 18 kB 00:00 2023-07-20T02:39:52.578 INFO:teuthology.orchestra.run.smithi158.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 120 kB/s | 20 kB 00:00 2023-07-20T02:39:52.619 INFO:teuthology.orchestra.run.smithi188.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 450 kB/s | 49 kB 00:00 2023-07-20T02:39:52.652 INFO:teuthology.orchestra.run.smithi158.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 500 kB/s | 103 kB 00:00 2023-07-20T02:39:52.726 INFO:teuthology.orchestra.run.smithi188.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 543 kB/s | 59 kB 00:00 2023-07-20T02:39:52.796 INFO:teuthology.orchestra.run.smithi158.stdout:(119/131): libtool-2.4.6-25.el8.x86_64.rpm 3.2 MB/s | 709 kB 00:00 2023-07-20T02:39:52.835 INFO:teuthology.orchestra.run.smithi188.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 556 kB/s | 60 kB 00:00 2023-07-20T02:39:52.946 INFO:teuthology.orchestra.run.smithi158.stdout:(120/131): openblas-threads-0.3.15-3.el8.x86_64 17 MB/s | 4.9 MB 00:00 2023-07-20T02:39:52.971 INFO:teuthology.orchestra.run.smithi158.stdout:(121/131): libpq-13.5-1.el8.x86_64.rpm 1.1 MB/s | 198 kB 00:00 2023-07-20T02:39:52.985 INFO:teuthology.orchestra.run.smithi188.stdout:(54/131): python3-pecan-1.3.2-9.el8.noarch.rpm 1.8 MB/s | 283 kB 00:00 2023-07-20T02:39:53.044 INFO:teuthology.orchestra.run.smithi188.stdout:(55/131): python3-kubernetes-11.0.0-6.el8.noarc 1.9 MB/s | 1.2 MB 00:00 2023-07-20T02:39:53.088 INFO:teuthology.orchestra.run.smithi158.stdout:(122/131): jq-1.6-3.el8.x86_64.rpm 1.7 MB/s | 202 kB 00:00 2023-07-20T02:39:53.114 INFO:teuthology.orchestra.run.smithi188.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 121 kB/s | 16 kB 00:00 2023-07-20T02:39:53.178 INFO:teuthology.orchestra.run.smithi188.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 252 kB/s | 34 kB 00:00 2023-07-20T02:39:53.197 INFO:teuthology.orchestra.run.smithi158.stdout:(123/131): openblas-0.3.15-3.el8.x86_64.rpm 19 MB/s | 4.7 MB 00:00 2023-07-20T02:39:53.222 INFO:teuthology.orchestra.run.smithi158.stdout:(124/131): librdkafka-0.11.4-3.el8.x86_64.rpm 2.6 MB/s | 354 kB 00:00 2023-07-20T02:39:53.262 INFO:teuthology.orchestra.run.smithi188.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 1.3 MB/s | 196 kB 00:00 2023-07-20T02:39:53.295 INFO:teuthology.orchestra.run.smithi188.stdout:(59/131): python3-rsa-4.8-1.el8.noarch.rpm 526 kB/s | 61 kB 00:00 2023-07-20T02:39:53.371 INFO:teuthology.orchestra.run.smithi188.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 172 kB/s | 19 kB 00:00 2023-07-20T02:39:53.412 INFO:teuthology.orchestra.run.smithi188.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 204 kB/s | 24 kB 00:00 2023-07-20T02:39:53.447 INFO:teuthology.orchestra.run.smithi158.stdout:(125/131): postgresql-server-10.21-2.module+el8 20 MB/s | 5.1 MB 00:00 2023-07-20T02:39:53.481 INFO:teuthology.orchestra.run.smithi158.stdout:(126/131): postgresql-contrib-10.21-2.module+el 3.1 MB/s | 808 kB 00:00 2023-07-20T02:39:53.496 INFO:teuthology.orchestra.run.smithi188.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 233 kB/s | 29 kB 00:00 2023-07-20T02:39:53.573 INFO:teuthology.orchestra.run.smithi158.stdout:(127/131): postgresql-10.21-2.module+el8.6.0+15 12 MB/s | 1.5 MB 00:00 2023-07-20T02:39:53.580 INFO:teuthology.orchestra.run.smithi188.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 165 kB/s | 27 kB 00:00 2023-07-20T02:39:53.621 INFO:teuthology.orchestra.run.smithi188.stdout:(64/131): python3-waitress-1.4.3-1.el8.noarch.r 2.0 MB/s | 255 kB 00:00 2023-07-20T02:39:53.722 INFO:teuthology.orchestra.run.smithi188.stdout:(65/131): python3-webob-1.8.5-1.el8.1.noarch.rp 1.7 MB/s | 251 kB 00:00 2023-07-20T02:39:53.747 INFO:teuthology.orchestra.run.smithi188.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 487 kB/s | 61 kB 00:00 2023-07-20T02:39:53.839 INFO:teuthology.orchestra.run.smithi188.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 731 kB/s | 85 kB 00:00 2023-07-20T02:39:53.865 INFO:teuthology.orchestra.run.smithi188.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 196 kB/s | 23 kB 00:00 2023-07-20T02:39:53.890 INFO:teuthology.orchestra.run.smithi188.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-07-20T02:39:54.082 INFO:teuthology.orchestra.run.smithi188.stdout:(70/131): thrift-0.13.0-2.el8.x86_64.rpm 7.2 MB/s | 1.7 MB 00:00 2023-07-20T02:39:54.107 INFO:teuthology.orchestra.run.smithi188.stdout:(71/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 216 kB/s | 47 kB 00:00 2023-07-20T02:39:54.193 INFO:teuthology.orchestra.run.smithi158.stdout:(128/131): thrift-0.13.0-2.el8.x86_64.rpm 217 kB/s | 1.7 MB 00:08 2023-07-20T02:39:54.208 INFO:teuthology.orchestra.run.smithi188.stdout:(72/131): python3-jwt-1.6.1-2.el8.noarch.rpm 452 kB/s | 43 kB 00:00 2023-07-20T02:39:54.241 INFO:teuthology.orchestra.run.smithi188.stdout:(73/131): libselinux-devel-2.9-5.el8.x86_64.rpm 1.2 MB/s | 200 kB 00:00 2023-07-20T02:39:54.342 INFO:teuthology.orchestra.run.smithi188.stdout:(74/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.6 MB/s | 220 kB 00:00 2023-07-20T02:39:54.365 INFO:teuthology.orchestra.run.smithi158.stdout:(129/131): librados2-18.0.0-5040.g1b0b2eb0.el8. 4.2 MB/s | 3.7 MB 00:00 2023-07-20T02:39:54.392 INFO:teuthology.orchestra.run.smithi188.stdout:(75/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 320 kB/s | 48 kB 00:00 2023-07-20T02:39:54.399 INFO:teuthology.orchestra.run.smithi158.stdout:(130/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 3.8 MB/s | 798 kB 00:00 2023-07-20T02:39:54.467 INFO:teuthology.orchestra.run.smithi188.stdout:(76/131): libsepol-devel-2.9-3.el8.x86_64.rpm 693 kB/s | 87 kB 00:00 2023-07-20T02:39:54.518 INFO:teuthology.orchestra.run.smithi188.stdout:(77/131): python3-requests-oauthlib-1.0.0-1.el8 340 kB/s | 43 kB 00:00 2023-07-20T02:39:54.609 INFO:teuthology.orchestra.run.smithi188.stdout:(78/131): python3-pycparser-2.14-14.el8.noarch. 769 kB/s | 109 kB 00:00 2023-07-20T02:39:54.649 INFO:teuthology.orchestra.run.smithi158.stdout:(131/131): librbd1-18.0.0-5040.g1b0b2eb0.el8.x8 3.4 MB/s | 3.6 MB 00:01 2023-07-20T02:39:54.651 INFO:teuthology.orchestra.run.smithi188.stdout:(79/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.7 MB/s | 229 kB 00:00 2023-07-20T02:39:54.653 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2023-07-20T02:39:54.653 INFO:teuthology.orchestra.run.smithi158.stdout:Total 8.2 MB/s | 206 MB 00:25 2023-07-20T02:39:54.802 INFO:teuthology.orchestra.run.smithi188.stdout:(80/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 2.9 MB/s | 560 kB 00:00 2023-07-20T02:39:54.883 INFO:teuthology.orchestra.run.smithi188.stdout:(81/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 81 kB/s | 19 kB 00:00 2023-07-20T02:39:54.969 INFO:teuthology.orchestra.run.smithi188.stdout:(82/131): python3-oauthlib-2.1.0-1.el8.noarch.r 928 kB/s | 155 kB 00:00 2023-07-20T02:39:55.011 INFO:teuthology.orchestra.run.smithi188.stdout:(83/131): python3-cryptography-3.2.1-5.el8.x86_ 4.3 MB/s | 559 kB 00:00 2023-07-20T02:39:55.111 INFO:teuthology.orchestra.run.smithi188.stdout:(84/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 126 kB/s | 18 kB 00:00 2023-07-20T02:39:55.178 INFO:teuthology.orchestra.run.smithi188.stdout:(85/131): pcre2-devel-10.32-2.el8.x86_64.rpm 3.5 MB/s | 605 kB 00:00 2023-07-20T02:39:55.270 INFO:teuthology.orchestra.run.smithi188.stdout:(86/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.5 MB/s | 238 kB 00:00 2023-07-20T02:39:55.362 INFO:teuthology.orchestra.run.smithi188.stdout:(87/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 13 MB/s | 2.3 MB 00:00 2023-07-20T02:39:55.446 INFO:teuthology.orchestra.run.smithi188.stdout:(88/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 412 kB/s | 72 kB 00:00 2023-07-20T02:39:55.480 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2023-07-20T02:39:55.505 INFO:teuthology.orchestra.run.smithi188.stdout:(89/131): fuse-2.9.7-15.el8.x86_64.rpm 583 kB/s | 83 kB 00:00 2023-07-20T02:39:55.601 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2023-07-20T02:39:55.601 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2023-07-20T02:39:55.630 INFO:teuthology.orchestra.run.smithi188.stdout:(90/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 784 kB/s | 98 kB 00:00 2023-07-20T02:39:55.655 INFO:teuthology.orchestra.run.smithi188.stdout:(91/131): elfutils-libelf-devel-0.186-1.el8.x86 286 kB/s | 60 kB 00:00 2023-07-20T02:39:56.481 INFO:teuthology.orchestra.run.smithi188.stdout:(92/131): ceph-test-18.0.0-5040.g1b0b2eb0.el8.x 2.6 MB/s | 48 MB 00:18 2023-07-20T02:39:56.506 INFO:teuthology.orchestra.run.smithi188.stdout:(93/131): libcom_err-devel-1.45.6-4.el8.x86_64. 45 kB/s | 39 kB 00:00 2023-07-20T02:39:56.540 INFO:teuthology.orchestra.run.smithi188.stdout:(94/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 63 kB/s | 58 kB 00:00 2023-07-20T02:39:56.640 INFO:teuthology.orchestra.run.smithi188.stdout:(95/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 1.3 MB/s | 171 kB 00:00 2023-07-20T02:39:56.657 INFO:teuthology.orchestra.run.smithi188.stdout:(96/131): NetworkManager-initscripts-updown-1.3 1.2 MB/s | 138 kB 00:00 2023-07-20T02:39:56.724 INFO:teuthology.orchestra.run.smithi188.stdout:(97/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 2.6 MB/s | 644 kB 00:00 2023-07-20T02:39:56.725 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2023-07-20T02:39:56.730 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2023-07-20T02:39:56.841 INFO:teuthology.orchestra.run.smithi188.stdout:(98/131): python3-pyasn1-modules-0.3.7-6.el8.no 600 kB/s | 110 kB 00:00 2023-07-20T02:39:56.908 INFO:teuthology.orchestra.run.smithi188.stdout:(99/131): python3-pytz-2017.2-9.el8.noarch.rpm 295 kB/s | 54 kB 00:00 2023-07-20T02:39:57.050 INFO:teuthology.orchestra.run.smithi188.stdout:(100/131): python3-scipy-1.0.0-21.module+el8.5. 33 MB/s | 14 MB 00:00 2023-07-20T02:39:57.075 INFO:teuthology.orchestra.run.smithi188.stdout:(101/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.1 MB/s | 259 kB 00:00 2023-07-20T02:39:57.151 INFO:teuthology.orchestra.run.smithi188.stdout:(102/131): socat-1.7.4.1-1.el8.x86_64.rpm 1.3 MB/s | 323 kB 00:00 2023-07-20T02:39:57.192 INFO:teuthology.orchestra.run.smithi188.stdout:(103/131): python3-markupsafe-0.23-19.el8.x86_6 273 kB/s | 39 kB 00:00 2023-07-20T02:39:57.218 INFO:teuthology.orchestra.run.smithi188.stdout:(104/131): bison-3.0.4-10.el8.x86_64.rpm 4.7 MB/s | 688 kB 00:00 2023-07-20T02:39:57.309 INFO:teuthology.orchestra.run.smithi188.stdout:(105/131): python3-jinja2-2.10.1-3.el8.noarch.r 3.3 MB/s | 538 kB 00:00 2023-07-20T02:39:57.351 INFO:teuthology.orchestra.run.smithi188.stdout:(106/131): python3-mako-1.0.6-13.el8.noarch.rpm 989 kB/s | 157 kB 00:00 2023-07-20T02:39:57.410 INFO:teuthology.orchestra.run.smithi188.stdout:(107/131): oniguruma-6.8.2-2.el8.x86_64.rpm 976 kB/s | 187 kB 00:00 2023-07-20T02:39:57.477 INFO:teuthology.orchestra.run.smithi188.stdout:(108/131): automake-1.16.1-7.el8.noarch.rpm 4.2 MB/s | 713 kB 00:00 2023-07-20T02:39:57.519 INFO:teuthology.orchestra.run.smithi188.stdout:(109/131): python3-jsonpatch-1.21-2.el8.noarch. 244 kB/s | 27 kB 00:00 2023-07-20T02:39:57.636 INFO:teuthology.orchestra.run.smithi188.stdout:(110/131): python3-babel-2.5.1-7.el8.noarch.rpm 17 MB/s | 4.8 MB 00:00 2023-07-20T02:39:57.661 INFO:teuthology.orchestra.run.smithi188.stdout:(111/131): flex-2.6.1-9.el8.x86_64.rpm 1.7 MB/s | 320 kB 00:00 2023-07-20T02:39:57.737 INFO:teuthology.orchestra.run.smithi188.stdout:(112/131): python3-prettytable-0.7.2-14.el8.noa 202 kB/s | 44 kB 00:00 2023-07-20T02:39:57.787 INFO:teuthology.orchestra.run.smithi188.stdout:(113/131): python3-werkzeug-0.12.2-4.el8.noarch 3.6 MB/s | 457 kB 00:00 2023-07-20T02:39:57.871 INFO:teuthology.orchestra.run.smithi188.stdout:(114/131): python3-numpy-1.14.3-10.el8.x86_64.r 16 MB/s | 3.7 MB 00:00 2023-07-20T02:39:57.896 INFO:teuthology.orchestra.run.smithi188.stdout:(115/131): python3-numpy-f2py-1.14.3-10.el8.x86 1.4 MB/s | 225 kB 00:00 2023-07-20T02:39:57.954 INFO:teuthology.orchestra.run.smithi188.stdout:(116/131): uuid-1.6.2-43.el8.x86_64.rpm 382 kB/s | 64 kB 00:00 2023-07-20T02:39:58.054 INFO:teuthology.orchestra.run.smithi188.stdout:(117/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 688 kB/s | 126 kB 00:00 2023-07-20T02:39:58.082 INFO:teuthology.orchestra.run.smithi188.stdout:(118/131): python3-jsonpointer-1.10-11.el8.noar 107 kB/s | 20 kB 00:00 2023-07-20T02:39:58.107 INFO:teuthology.orchestra.run.smithi188.stdout:(119/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 678 kB/s | 103 kB 00:00 2023-07-20T02:39:58.216 INFO:teuthology.orchestra.run.smithi188.stdout:(120/131): libtool-2.4.6-25.el8.x86_64.rpm 4.3 MB/s | 709 kB 00:00 2023-07-20T02:39:58.266 INFO:teuthology.orchestra.run.smithi188.stdout:(121/131): libpq-13.5-1.el8.x86_64.rpm 1.2 MB/s | 198 kB 00:00 2023-07-20T02:39:58.350 INFO:teuthology.orchestra.run.smithi188.stdout:(122/131): openblas-threads-0.3.15-3.el8.x86_64 18 MB/s | 4.9 MB 00:00 2023-07-20T02:39:58.417 INFO:teuthology.orchestra.run.smithi188.stdout:(123/131): jq-1.6-3.el8.x86_64.rpm 1.3 MB/s | 202 kB 00:00 2023-07-20T02:39:58.517 INFO:teuthology.orchestra.run.smithi188.stdout:(124/131): openblas-0.3.15-3.el8.x86_64.rpm 16 MB/s | 4.7 MB 00:00 2023-07-20T02:39:58.542 INFO:teuthology.orchestra.run.smithi188.stdout:(125/131): librdkafka-0.11.4-3.el8.x86_64.rpm 1.8 MB/s | 354 kB 00:00 2023-07-20T02:39:58.609 INFO:teuthology.orchestra.run.smithi188.stdout:(126/131): postgresql-contrib-10.21-2.module+el 8.6 MB/s | 808 kB 00:00 2023-07-20T02:39:58.693 INFO:teuthology.orchestra.run.smithi188.stdout:(127/131): postgresql-10.21-2.module+el8.6.0+15 10 MB/s | 1.5 MB 00:00 2023-07-20T02:39:58.793 INFO:teuthology.orchestra.run.smithi188.stdout:(128/131): postgresql-server-10.21-2.module+el8 13 MB/s | 5.1 MB 00:00 2023-07-20T02:39:58.968 INFO:teuthology.orchestra.run.smithi188.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 4.5 MB/s | 798 kB 00:00 2023-07-20T02:40:01.160 INFO:teuthology.orchestra.run.smithi188.stdout:(130/131): librbd1-18.0.0-5040.g1b0b2eb0.el8.x8 1.5 MB/s | 3.6 MB 00:02 2023-07-20T02:40:01.752 INFO:teuthology.orchestra.run.smithi188.stdout:(131/131): librados2-18.0.0-5040.g1b0b2eb0.el8. 1.2 MB/s | 3.7 MB 00:03 2023-07-20T02:40:01.756 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2023-07-20T02:40:01.756 INFO:teuthology.orchestra.run.smithi188.stdout:Total 6.4 MB/s | 206 MB 00:32 2023-07-20T02:40:02.320 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2023-07-20T02:40:02.542 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-20T02:40:02.574 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2023-07-20T02:40:02.695 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2023-07-20T02:40:02.695 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2023-07-20T02:40:02.734 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-20T02:40:02.961 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-20T02:40:03.116 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-20T02:40:03.378 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-20T02:40:03.783 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-20T02:40:03.827 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2023-07-20T02:40:03.832 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2023-07-20T02:40:04.060 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 5/134 2023-07-20T02:40:04.470 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 5/134 2023-07-20T02:40:04.685 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T02:40:04.800 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T02:40:04.904 INFO:teuthology.orchestra.run.smithi007.stdout:Extra Packages for Enterprise Linux 260 kB/s | 16 MB 01:01 2023-07-20T02:40:04.977 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T02:40:05.130 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T02:40:05.324 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T02:40:05.486 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-07-20T02:40:06.030 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-20T02:40:06.257 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-20T02:40:06.677 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-20T02:40:06.939 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-20T02:40:07.118 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T02:40:07.220 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T02:40:07.410 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T02:40:07.545 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T02:40:07.797 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-07-20T02:40:08.008 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-07-20T02:40:08.219 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-07-20T02:40:08.393 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-20T02:40:08.548 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-20T02:40:08.751 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-20T02:40:08.821 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2023-07-20T02:40:08.985 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-20T02:40:09.118 INFO:teuthology.orchestra.run.smithi007.stdout:lab-extras 439 kB/s | 24 kB 00:00 2023-07-20T02:40:09.202 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-20T02:40:09.357 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-20T02:40:09.371 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-20T02:40:09.487 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-20T02:40:09.510 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-20T02:40:09.721 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-20T02:40:10.091 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-20T02:40:10.112 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-20T02:40:10.276 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-07-20T02:40:10.311 INFO:teuthology.orchestra.run.smithi007.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 47 MB/s | 53 MB 00:01 2023-07-20T02:40:10.313 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 5/134 2023-07-20T02:40:10.696 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 5/134 2023-07-20T02:40:10.929 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T02:40:11.028 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-07-20T02:40:11.051 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T02:40:11.237 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T02:40:11.330 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T02:40:11.392 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-07-20T02:40:11.493 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T02:40:11.584 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-07-20T02:40:11.670 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-07-20T02:40:11.914 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-20T02:40:11.924 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-20T02:40:12.107 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-20T02:40:12.141 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 26/134 2023-07-20T02:40:12.255 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 27/134 2023-07-20T02:40:12.509 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-20T02:40:12.529 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 28/134 2023-07-20T02:40:12.759 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-20T02:40:12.774 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-20T02:40:12.920 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T02:40:13.021 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T02:40:13.212 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T02:40:13.356 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T02:40:13.615 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-07-20T02:40:13.768 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-07-20T02:40:13.943 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-07-20T02:40:14.103 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-20T02:40:14.238 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-20T02:40:14.386 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-20T02:40:14.546 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-20T02:40:15.038 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-20T02:40:15.195 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-20T02:40:15.665 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-07-20T02:40:15.795 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-20T02:40:15.986 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-07-20T02:40:16.049 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-07-20T02:40:16.371 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-07-20T02:40:16.625 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-07-20T02:40:16.680 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-07-20T02:40:16.854 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-20T02:40:16.901 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-07-20T02:40:17.074 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-07-20T02:40:17.094 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-20T02:40:17.279 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-20T02:40:17.306 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 36/134 2023-07-20T02:40:17.418 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 26/134 2023-07-20T02:40:17.507 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 37/134 2023-07-20T02:40:17.581 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 27/134 2023-07-20T02:40:17.708 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-07-20T02:40:17.790 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 28/134 2023-07-20T02:40:17.863 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-07-20T02:40:18.011 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-20T02:40:18.092 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-07-20T02:40:18.161 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-07-20T02:40:18.420 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-07-20T02:40:18.662 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-07-20T02:40:18.858 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-20T02:40:18.976 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-20T02:40:19.198 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-07-20T02:40:19.379 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-07-20T02:40:19.592 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-07-20T02:40:19.780 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-20T02:40:19.786 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-07-20T02:40:19.813 INFO:teuthology.orchestra.run.smithi007.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 36 MB/s | 47 MB 00:01 2023-07-20T02:40:19.994 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-07-20T02:40:20.112 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-07-20T02:40:20.467 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-07-20T02:40:20.716 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-07-20T02:40:20.795 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-07-20T02:40:21.092 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-07-20T02:40:21.143 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-07-20T02:40:21.379 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-07-20T02:40:21.518 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-07-20T02:40:21.648 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-07-20T02:40:21.803 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-07-20T02:40:21.830 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-20T02:40:22.037 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-20T02:40:22.065 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-07-20T02:40:22.199 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 36/134 2023-07-20T02:40:22.347 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-07-20T02:40:22.349 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 37/134 2023-07-20T02:40:22.525 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-07-20T02:40:22.652 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-07-20T02:40:22.685 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-07-20T02:40:22.818 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-07-20T02:40:22.922 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-07-20T02:40:22.958 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-07-20T02:40:23.129 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-07-20T02:40:23.204 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-07-20T02:40:23.388 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-07-20T02:40:23.491 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-07-20T02:40:23.526 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-20T02:40:23.609 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-20T02:40:23.705 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-07-20T02:40:23.839 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-07-20T02:40:23.870 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-07-20T02:40:23.971 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-07-20T02:40:24.125 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-07-20T02:40:24.206 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-07-20T02:40:24.319 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-07-20T02:40:24.487 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-07-20T02:40:24.508 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-07-20T02:40:24.612 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-07-20T02:40:24.796 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-07-20T02:40:24.829 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-07-20T02:40:25.003 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-07-20T02:40:25.067 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-07-20T02:40:25.283 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-20T02:40:25.383 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-07-20T02:40:25.521 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-20T02:40:25.711 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-07-20T02:40:25.839 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-20T02:40:25.954 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-07-20T02:40:26.115 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-20T02:40:26.181 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-07-20T02:40:26.304 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-20T02:40:26.405 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-07-20T02:40:26.503 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-20T02:40:26.717 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-07-20T02:40:26.740 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-20T02:40:26.958 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-07-20T02:40:26.996 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-20T02:40:27.135 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-07-20T02:40:27.241 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-20T02:40:27.388 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-07-20T02:40:27.469 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-20T02:40:27.529 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-07-20T02:40:27.597 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-20T02:40:27.687 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-07-20T02:40:27.904 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-20T02:40:28.024 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-07-20T02:40:28.236 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-20T02:40:28.322 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-07-20T02:40:28.488 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-20T02:40:28.532 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-07-20T02:40:28.746 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-20T02:40:28.793 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-07-20T02:40:28.933 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-20T02:40:29.023 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-20T02:40:29.144 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-20T02:40:29.248 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-20T02:40:29.358 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-20T02:40:29.525 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-20T02:40:29.623 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-20T02:40:29.770 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-20T02:40:29.910 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-20T02:40:30.131 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-20T02:40:30.318 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-20T02:40:30.525 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-20T02:40:30.677 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-20T02:40:30.702 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-20T02:40:30.797 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-20T02:40:31.019 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-20T02:40:31.041 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-20T02:40:31.243 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-20T02:40:31.316 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-20T02:40:31.455 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-20T02:40:31.680 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-20T02:40:31.685 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-20T02:40:31.915 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-20T02:40:32.048 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-20T02:40:32.072 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-20T02:40:32.181 INFO:teuthology.orchestra.run.smithi007.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T02:40:32.182 INFO:teuthology.orchestra.run.smithi007.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T02:40:32.198 INFO:teuthology.orchestra.run.smithi007.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-20T02:40:32.199 INFO:teuthology.orchestra.run.smithi007.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-20T02:40:32.201 INFO:teuthology.orchestra.run.smithi007.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-20T02:40:32.202 INFO:teuthology.orchestra.run.smithi007.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-20T02:40:32.204 INFO:teuthology.orchestra.run.smithi007.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-20T02:40:32.208 INFO:teuthology.orchestra.run.smithi007.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-20T02:40:32.209 INFO:teuthology.orchestra.run.smithi007.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-20T02:40:32.270 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-20T02:40:32.293 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-20T02:40:32.359 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-07-20T02:40:32.373 INFO:teuthology.orchestra.run.smithi007.stdout:=========================================================================================================================== 2023-07-20T02:40:32.373 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2023-07-20T02:40:32.373 INFO:teuthology.orchestra.run.smithi007.stdout:=========================================================================================================================== 2023-07-20T02:40:32.374 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-07-20T02:40:32.374 INFO:teuthology.orchestra.run.smithi007.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-20T02:40:32.374 INFO:teuthology.orchestra.run.smithi007.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-20T02:40:32.374 INFO:teuthology.orchestra.run.smithi007.stdout: ceph x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.4 k 2023-07-20T02:40:32.374 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 5.4 M 2023-07-20T02:40:32.374 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 850 k 2023-07-20T02:40:32.374 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 167 k 2023-07-20T02:40:32.375 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 1.6 M 2023-07-20T02:40:32.375 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 129 k 2023-07-20T02:40:32.375 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-dashboard noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 1.7 M 2023-07-20T02:40:32.375 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 7.4 M 2023-07-20T02:40:32.375 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-rook noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 47 k 2023-07-20T02:40:32.375 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 14 M 2023-07-20T02:40:32.376 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 48 M 2023-07-20T02:40:32.376 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-volume noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 258 k 2023-07-20T02:40:32.376 INFO:teuthology.orchestra.run.smithi007.stdout: cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 165 k 2023-07-20T02:40:32.376 INFO:teuthology.orchestra.run.smithi007.stdout: cephfs-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 260 k 2023-07-20T02:40:32.376 INFO:teuthology.orchestra.run.smithi007.stdout: cephfs-top noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 18 k 2023-07-20T02:40:32.376 INFO:teuthology.orchestra.run.smithi007.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-20T02:40:32.376 INFO:teuthology.orchestra.run.smithi007.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-20T02:40:32.376 INFO:teuthology.orchestra.run.smithi007.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-20T02:40:32.377 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 31 k 2023-07-20T02:40:32.377 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 771 k 2023-07-20T02:40:32.377 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T02:40:32.377 INFO:teuthology.orchestra.run.smithi007.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-20T02:40:32.377 INFO:teuthology.orchestra.run.smithi007.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-20T02:40:32.377 INFO:teuthology.orchestra.run.smithi007.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-20T02:40:32.377 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-20T02:40:32.378 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-20T02:40:32.378 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-20T02:40:32.378 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 197 k 2023-07-20T02:40:32.378 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 370 k 2023-07-20T02:40:32.378 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 363 k 2023-07-20T02:40:32.378 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 107 k 2023-07-20T02:40:32.379 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 94 k 2023-07-20T02:40:32.379 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.5 M 2023-07-20T02:40:32.379 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 179 k 2023-07-20T02:40:32.379 INFO:teuthology.orchestra.run.smithi007.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-20T02:40:32.379 INFO:teuthology.orchestra.run.smithi007.stdout:Upgrading: 2023-07-20T02:40:32.379 INFO:teuthology.orchestra.run.smithi007.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-20T02:40:32.379 INFO:teuthology.orchestra.run.smithi007.stdout: librados2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.7 M 2023-07-20T02:40:32.379 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.6 M 2023-07-20T02:40:32.380 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2023-07-20T02:40:32.380 INFO:teuthology.orchestra.run.smithi007.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-20T02:40:32.380 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 22 M 2023-07-20T02:40:32.380 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-grafana-dashboards noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 23 k 2023-07-20T02:40:32.380 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 2.3 M 2023-07-20T02:40:32.380 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-modules-core noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 241 k 2023-07-20T02:40:32.380 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 4.6 M 2023-07-20T02:40:32.381 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 18 M 2023-07-20T02:40:32.381 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-prometheus-alerts noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 15 k 2023-07-20T02:40:32.381 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-selinux x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 24 k 2023-07-20T02:40:32.381 INFO:teuthology.orchestra.run.smithi007.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-20T02:40:32.381 INFO:teuthology.orchestra.run.smithi007.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-20T02:40:32.381 INFO:teuthology.orchestra.run.smithi007.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-20T02:40:32.381 INFO:teuthology.orchestra.run.smithi007.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-20T02:40:32.382 INFO:teuthology.orchestra.run.smithi007.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-20T02:40:32.383 INFO:teuthology.orchestra.run.smithi007.stdout: libcephsqlite x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T02:40:32.383 INFO:teuthology.orchestra.run.smithi007.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-20T02:40:32.383 INFO:teuthology.orchestra.run.smithi007.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-20T02:40:32.384 INFO:teuthology.orchestra.run.smithi007.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-20T02:40:32.384 INFO:teuthology.orchestra.run.smithi007.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-20T02:40:32.384 INFO:teuthology.orchestra.run.smithi007.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-20T02:40:32.385 INFO:teuthology.orchestra.run.smithi007.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-20T02:40:32.385 INFO:teuthology.orchestra.run.smithi007.stdout: libradosstriper1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 472 k 2023-07-20T02:40:32.385 INFO:teuthology.orchestra.run.smithi007.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-20T02:40:32.385 INFO:teuthology.orchestra.run.smithi007.stdout: librgw2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.8 M 2023-07-20T02:40:32.386 INFO:teuthology.orchestra.run.smithi007.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-20T02:40:32.386 INFO:teuthology.orchestra.run.smithi007.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-20T02:40:32.387 INFO:teuthology.orchestra.run.smithi007.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-20T02:40:32.387 INFO:teuthology.orchestra.run.smithi007.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-20T02:40:32.387 INFO:teuthology.orchestra.run.smithi007.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-20T02:40:32.387 INFO:teuthology.orchestra.run.smithi007.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-20T02:40:32.388 INFO:teuthology.orchestra.run.smithi007.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-20T02:40:32.388 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-20T02:40:32.388 INFO:teuthology.orchestra.run.smithi007.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-20T02:40:32.389 INFO:teuthology.orchestra.run.smithi007.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-20T02:40:32.389 INFO:teuthology.orchestra.run.smithi007.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-20T02:40:32.389 INFO:teuthology.orchestra.run.smithi007.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-20T02:40:32.390 INFO:teuthology.orchestra.run.smithi007.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-20T02:40:32.390 INFO:teuthology.orchestra.run.smithi007.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-20T02:40:32.390 INFO:teuthology.orchestra.run.smithi007.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-20T02:40:32.390 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-20T02:40:32.391 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-argparse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 45 k 2023-07-20T02:40:32.391 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 122 k 2023-07-20T02:40:32.391 INFO:teuthology.orchestra.run.smithi007.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-20T02:40:32.391 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-20T02:40:32.391 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-20T02:40:32.392 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-20T02:40:32.392 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-20T02:40:32.392 INFO:teuthology.orchestra.run.smithi007.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-20T02:40:32.393 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-20T02:40:32.393 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-20T02:40:32.393 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-20T02:40:32.394 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-20T02:40:32.394 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-20T02:40:32.394 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T02:40:32.394 INFO:teuthology.orchestra.run.smithi007.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-20T02:40:32.395 INFO:teuthology.orchestra.run.smithi007.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-20T02:40:32.395 INFO:teuthology.orchestra.run.smithi007.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-20T02:40:32.395 INFO:teuthology.orchestra.run.smithi007.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-20T02:40:32.396 INFO:teuthology.orchestra.run.smithi007.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-20T02:40:32.396 INFO:teuthology.orchestra.run.smithi007.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-20T02:40:32.396 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-20T02:40:32.396 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-20T02:40:32.397 INFO:teuthology.orchestra.run.smithi007.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-20T02:40:32.397 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-20T02:40:32.397 INFO:teuthology.orchestra.run.smithi007.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-20T02:40:32.398 INFO:teuthology.orchestra.run.smithi007.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-20T02:40:32.398 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-20T02:40:32.398 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-20T02:40:32.398 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-20T02:40:32.398 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-20T02:40:32.399 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-20T02:40:32.399 INFO:teuthology.orchestra.run.smithi007.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-20T02:40:32.399 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T02:40:32.399 INFO:teuthology.orchestra.run.smithi007.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-20T02:40:32.400 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-20T02:40:32.400 INFO:teuthology.orchestra.run.smithi007.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-20T02:40:32.400 INFO:teuthology.orchestra.run.smithi007.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-20T02:40:32.400 INFO:teuthology.orchestra.run.smithi007.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-20T02:40:32.401 INFO:teuthology.orchestra.run.smithi007.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-20T02:40:32.401 INFO:teuthology.orchestra.run.smithi007.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-20T02:40:32.401 INFO:teuthology.orchestra.run.smithi007.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-20T02:40:32.402 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-20T02:40:32.402 INFO:teuthology.orchestra.run.smithi007.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-20T02:40:32.402 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-20T02:40:32.402 INFO:teuthology.orchestra.run.smithi007.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-20T02:40:32.402 INFO:teuthology.orchestra.run.smithi007.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-20T02:40:32.403 INFO:teuthology.orchestra.run.smithi007.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-20T02:40:32.403 INFO:teuthology.orchestra.run.smithi007.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-20T02:40:32.403 INFO:teuthology.orchestra.run.smithi007.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-20T02:40:32.404 INFO:teuthology.orchestra.run.smithi007.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-20T02:40:32.405 INFO:teuthology.orchestra.run.smithi007.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-20T02:40:32.405 INFO:teuthology.orchestra.run.smithi007.stdout:Enabling module streams: 2023-07-20T02:40:32.405 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql 10 2023-07-20T02:40:32.405 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:40:32.405 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-07-20T02:40:32.405 INFO:teuthology.orchestra.run.smithi007.stdout:=========================================================================================================================== 2023-07-20T02:40:32.406 INFO:teuthology.orchestra.run.smithi007.stdout:Install 128 Packages 2023-07-20T02:40:32.406 INFO:teuthology.orchestra.run.smithi007.stdout:Upgrade 3 Packages 2023-07-20T02:40:32.406 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:40:32.407 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 206 M 2023-07-20T02:40:32.407 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-07-20T02:40:32.664 INFO:teuthology.orchestra.run.smithi007.stdout:(1/131): ceph-18.0.0-5040.g1b0b2eb0.el8.x86_64. 27 kB/s | 6.4 kB 00:00 2023-07-20T02:40:33.139 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-20T02:40:33.305 INFO:teuthology.orchestra.run.smithi007.stdout:(2/131): ceph-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 1.3 MB/s | 850 kB 00:00 2023-07-20T02:40:33.356 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T02:40:33.429 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T02:40:33.488 INFO:teuthology.orchestra.run.smithi007.stdout:(3/131): ceph-immutable-object-cache-18.0.0-504 912 kB/s | 167 kB 00:00 2023-07-20T02:40:33.656 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 93/134 2023-07-20T02:40:33.732 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-20T02:40:33.994 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-20T02:40:34.265 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-20T02:40:34.272 INFO:teuthology.orchestra.run.smithi007.stdout:(4/131): ceph-mds-18.0.0-5040.g1b0b2eb0.el8.x86 2.9 MB/s | 2.3 MB 00:00 2023-07-20T02:40:34.541 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-20T02:40:34.547 INFO:teuthology.orchestra.run.smithi007.stdout:(5/131): ceph-base-18.0.0-5040.g1b0b2eb0.el8.x8 2.5 MB/s | 5.4 MB 00:02 2023-07-20T02:40:34.747 INFO:teuthology.orchestra.run.smithi007.stdout:(6/131): ceph-mgr-18.0.0-5040.g1b0b2eb0.el8.x86 3.3 MB/s | 1.6 MB 00:00 2023-07-20T02:40:34.804 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-20T02:40:35.035 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-20T02:40:35.423 INFO:teuthology.orchestra.run.smithi007.stdout:(7/131): ceph-mon-18.0.0-5040.g1b0b2eb0.el8.x86 5.3 MB/s | 4.6 MB 00:00 2023-07-20T02:40:35.817 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-20T02:40:35.982 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T02:40:36.052 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T02:40:36.216 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 93/134 2023-07-20T02:40:36.398 INFO:teuthology.orchestra.run.smithi007.stdout:(8/131): ceph-common-18.0.0-5040.g1b0b2eb0.el8. 5.6 MB/s | 22 MB 00:03 2023-07-20T02:40:36.473 INFO:teuthology.orchestra.run.smithi007.stdout:(9/131): ceph-selinux-18.0.0-5040.g1b0b2eb0.el8 325 kB/s | 24 kB 00:00 2023-07-20T02:40:37.531 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T02:40:37.873 INFO:teuthology.orchestra.run.smithi007.stdout:(10/131): ceph-radosgw-18.0.0-5040.g1b0b2eb0.el 5.5 MB/s | 14 MB 00:02 2023-07-20T02:40:37.965 INFO:teuthology.orchestra.run.smithi007.stdout:(11/131): cephfs-mirror-18.0.0-5040.g1b0b2eb0.e 2.8 MB/s | 260 kB 00:00 2023-07-20T02:40:38.015 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T02:40:38.049 INFO:teuthology.orchestra.run.smithi007.stdout:(12/131): libcephfs-devel-18.0.0-5040.g1b0b2eb0 373 kB/s | 31 kB 00:00 2023-07-20T02:40:38.257 INFO:teuthology.orchestra.run.smithi007.stdout:(13/131): libcephfs2-18.0.0-5040.g1b0b2eb0.el8. 3.6 MB/s | 771 kB 00:00 2023-07-20T02:40:38.499 INFO:teuthology.orchestra.run.smithi007.stdout:(14/131): ceph-osd-18.0.0-5040.g1b0b2eb0.el8.x8 4.7 MB/s | 18 MB 00:03 2023-07-20T02:40:38.525 INFO:teuthology.orchestra.run.smithi007.stdout:(15/131): libcephsqlite-18.0.0-5040.g1b0b2eb0.e 520 kB/s | 139 kB 00:00 2023-07-20T02:40:38.583 INFO:teuthology.orchestra.run.smithi007.stdout:(16/131): librados-devel-18.0.0-5040.g1b0b2eb0. 1.6 MB/s | 139 kB 00:00 2023-07-20T02:40:38.594 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T02:40:38.667 INFO:teuthology.orchestra.run.smithi007.stdout:(17/131): libradosstriper1-18.0.0-5040.g1b0b2eb 3.3 MB/s | 472 kB 00:00 2023-07-20T02:40:38.759 INFO:teuthology.orchestra.run.smithi007.stdout:(18/131): python3-ceph-argparse-18.0.0-5040.g1b 488 kB/s | 45 kB 00:00 2023-07-20T02:40:38.859 INFO:teuthology.orchestra.run.smithi007.stdout:(19/131): python3-ceph-common-18.0.0-5040.g1b0b 1.2 MB/s | 122 kB 00:00 2023-07-20T02:40:38.911 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T02:40:38.968 INFO:teuthology.orchestra.run.smithi007.stdout:(20/131): python3-cephfs-18.0.0-5040.g1b0b2eb0. 1.8 MB/s | 197 kB 00:00 2023-07-20T02:40:39.002 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T02:40:39.118 INFO:teuthology.orchestra.run.smithi007.stdout:(21/131): python3-rados-18.0.0-5040.g1b0b2eb0.e 2.4 MB/s | 370 kB 00:00 2023-07-20T02:40:39.197 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T02:40:39.260 INFO:teuthology.orchestra.run.smithi007.stdout:(22/131): python3-rbd-18.0.0-5040.g1b0b2eb0.el8 2.5 MB/s | 363 kB 00:00 2023-07-20T02:40:39.344 INFO:teuthology.orchestra.run.smithi007.stdout:(23/131): python3-rgw-18.0.0-5040.g1b0b2eb0.el8 1.2 MB/s | 107 kB 00:00 2023-07-20T02:40:39.444 INFO:teuthology.orchestra.run.smithi007.stdout:(24/131): rbd-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 939 kB/s | 94 kB 00:00 2023-07-20T02:40:39.903 INFO:teuthology.orchestra.run.smithi007.stdout:(25/131): ceph-test-18.0.0-5040.g1b0b2eb0.el8.x 14 MB/s | 48 MB 00:03 2023-07-20T02:40:39.914 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T02:40:39.986 INFO:teuthology.orchestra.run.smithi007.stdout:(26/131): rbd-nbd-18.0.0-5040.g1b0b2eb0.el8.x86 2.1 MB/s | 179 kB 00:00 2023-07-20T02:40:40.095 INFO:teuthology.orchestra.run.smithi007.stdout:(27/131): ceph-grafana-dashboards-18.0.0-5040.g 213 kB/s | 23 kB 00:00 2023-07-20T02:40:40.107 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T02:40:40.237 INFO:teuthology.orchestra.run.smithi007.stdout:(28/131): librgw2-18.0.0-5040.g1b0b2eb0.el8.x86 4.1 MB/s | 6.8 MB 00:01 2023-07-20T02:40:40.262 INFO:teuthology.orchestra.run.smithi007.stdout:(29/131): ceph-mgr-cephadm-18.0.0-5040.g1b0b2eb 775 kB/s | 129 kB 00:00 2023-07-20T02:40:40.329 INFO:teuthology.orchestra.run.smithi007.stdout:(30/131): rbd-mirror-18.0.0-5040.g1b0b2eb0.el8. 4.0 MB/s | 3.5 MB 00:00 2023-07-20T02:40:40.430 INFO:teuthology.orchestra.run.smithi007.stdout:(31/131): ceph-mgr-modules-core-18.0.0-5040.g1b 2.4 MB/s | 241 kB 00:00 2023-07-20T02:40:40.513 INFO:teuthology.orchestra.run.smithi007.stdout:(32/131): ceph-mgr-rook-18.0.0-5040.g1b0b2eb0.e 572 kB/s | 47 kB 00:00 2023-07-20T02:40:40.597 INFO:teuthology.orchestra.run.smithi007.stdout:(33/131): ceph-mgr-dashboard-18.0.0-5040.g1b0b2 4.8 MB/s | 1.7 MB 00:00 2023-07-20T02:40:40.644 INFO:teuthology.orchestra.run.smithi007.stdout:(34/131): ceph-prometheus-alerts-18.0.0-5040.g1 112 kB/s | 15 kB 00:00 2023-07-20T02:40:40.685 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T02:40:40.697 INFO:teuthology.orchestra.run.smithi007.stdout:(35/131): ceph-volume-18.0.0-5040.g1b0b2eb0.el8 2.5 MB/s | 258 kB 00:00 2023-07-20T02:40:40.724 INFO:teuthology.orchestra.run.smithi007.stdout:(36/131): cephadm-18.0.0-5040.g1b0b2eb0.el8.noa 2.1 MB/s | 165 kB 00:00 2023-07-20T02:40:40.773 INFO:teuthology.orchestra.run.smithi007.stdout:(37/131): cephfs-top-18.0.0-5040.g1b0b2eb0.el8. 239 kB/s | 18 kB 00:00 2023-07-20T02:40:40.898 INFO:teuthology.orchestra.run.smithi007.stdout:(38/131): ceph-mgr-diskprediction-local-18.0.0- 12 MB/s | 7.4 MB 00:00 2023-07-20T02:40:40.992 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T02:40:41.079 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T02:40:41.123 INFO:teuthology.orchestra.run.smithi007.stdout:(39/131): liboath-2.6.2-3.el8.x86_64.rpm 170 kB/s | 59 kB 00:00 2023-07-20T02:40:41.157 INFO:teuthology.orchestra.run.smithi007.stdout:(40/131): libunwind-1.3.1-3.el8.x86_64.rpm 297 kB/s | 75 kB 00:00 2023-07-20T02:40:41.191 INFO:teuthology.orchestra.run.smithi007.stdout:(41/131): gperftools-libs-2.7-9.el8.x86_64.rpm 655 kB/s | 306 kB 00:00 2023-07-20T02:40:41.258 INFO:teuthology.orchestra.run.smithi007.stdout:(42/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 434 kB/s | 44 kB 00:00 2023-07-20T02:40:41.281 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T02:40:41.325 INFO:teuthology.orchestra.run.smithi007.stdout:(43/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.4 MB/s | 185 kB 00:00 2023-07-20T02:40:41.350 INFO:teuthology.orchestra.run.smithi007.stdout:(44/131): python3-cachetools-3.1.1-4.el8.noarch 361 kB/s | 33 kB 00:00 2023-07-20T02:40:41.392 INFO:teuthology.orchestra.run.smithi007.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 245 kB/s | 16 kB 00:00 2023-07-20T02:40:41.425 INFO:teuthology.orchestra.run.smithi007.stdout:(46/131): python3-asyncssh-2.7.0-2.el8.noarch.r 1.5 MB/s | 476 kB 00:00 2023-07-20T02:40:41.459 INFO:teuthology.orchestra.run.smithi007.stdout:(47/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.5 MB/s | 173 kB 00:00 2023-07-20T02:40:41.501 INFO:teuthology.orchestra.run.smithi007.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 1.1 MB/s | 81 kB 00:00 2023-07-20T02:40:41.518 INFO:teuthology.orchestra.run.smithi007.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 188 kB/s | 11 kB 00:00 2023-07-20T02:40:41.568 INFO:teuthology.orchestra.run.smithi007.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 263 kB/s | 18 kB 00:00 2023-07-20T02:40:41.594 INFO:teuthology.orchestra.run.smithi007.stdout:(51/131): python3-cherrypy-18.4.0-1.el8.noarch. 1.9 MB/s | 384 kB 00:00 2023-07-20T02:40:41.661 INFO:teuthology.orchestra.run.smithi007.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 883 kB/s | 59 kB 00:00 2023-07-20T02:40:41.861 INFO:teuthology.orchestra.run.smithi007.stdout:(53/131): python3-logutils-0.3.5-11.el8.noarch. 167 kB/s | 49 kB 00:00 2023-07-20T02:40:41.953 INFO:teuthology.orchestra.run.smithi007.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 206 kB/s | 60 kB 00:00 2023-07-20T02:40:42.028 INFO:teuthology.orchestra.run.smithi007.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 208 kB/s | 16 kB 00:00 2023-07-20T02:40:42.062 INFO:teuthology.orchestra.run.smithi007.stdout:(56/131): python3-pecan-1.3.2-9.el8.noarch.rpm 1.4 MB/s | 283 kB 00:00 2023-07-20T02:40:42.096 INFO:teuthology.orchestra.run.smithi007.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 504 kB/s | 34 kB 00:00 2023-07-20T02:40:42.154 INFO:teuthology.orchestra.run.smithi007.stdout:(58/131): python3-kubernetes-11.0.0-6.el8.noarc 1.8 MB/s | 1.2 MB 00:00 2023-07-20T02:40:42.180 INFO:teuthology.orchestra.run.smithi007.stdout:(59/131): python3-rsa-4.8-1.el8.noarch.rpm 735 kB/s | 61 kB 00:00 2023-07-20T02:40:42.406 INFO:teuthology.orchestra.run.smithi007.stdout:(60/131): python3-routes-2.4.1-12.el8.noarch.rp 572 kB/s | 196 kB 00:00 2023-07-20T02:40:42.472 INFO:teuthology.orchestra.run.smithi007.stdout:(61/131): python3-simplegeneric-0.8.1-17.el8.no 59 kB/s | 19 kB 00:00 2023-07-20T02:40:42.497 INFO:teuthology.orchestra.run.smithi007.stdout:(62/131): python3-singledispatch-3.4.0.3-18.el8 75 kB/s | 24 kB 00:00 2023-07-20T02:40:42.522 INFO:teuthology.orchestra.run.smithi007.stdout:(63/131): python3-tempora-1.14.1-5.el8.noarch.r 252 kB/s | 29 kB 00:00 2023-07-20T02:40:42.573 INFO:teuthology.orchestra.run.smithi007.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 274 kB/s | 27 kB 00:00 2023-07-20T02:40:42.840 INFO:teuthology.orchestra.run.smithi007.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 229 kB/s | 61 kB 00:00 2023-07-20T02:40:42.890 INFO:teuthology.orchestra.run.smithi007.stdout:(66/131): python3-webob-1.8.5-1.el8.1.noarch.rp 684 kB/s | 251 kB 00:00 2023-07-20T02:40:42.965 INFO:teuthology.orchestra.run.smithi007.stdout:(67/131): python3-zc-lockfile-2.0-2.el8.noarch. 307 kB/s | 23 kB 00:00 2023-07-20T02:40:43.182 INFO:teuthology.orchestra.run.smithi007.stdout:(68/131): python3-waitress-1.4.3-1.el8.noarch.r 373 kB/s | 255 kB 00:00 2023-07-20T02:40:43.208 INFO:teuthology.orchestra.run.smithi007.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-07-20T02:40:43.283 INFO:teuthology.orchestra.run.smithi007.stdout:(70/131): python3-webtest-2.0.33-1.el8.noarch.r 193 kB/s | 85 kB 00:00 2023-07-20T02:40:43.383 INFO:teuthology.orchestra.run.smithi007.stdout:(71/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 267 kB/s | 47 kB 00:00 2023-07-20T02:40:43.450 INFO:teuthology.orchestra.run.smithi007.stdout:(72/131): libselinux-devel-2.9-5.el8.x86_64.rpm 1.2 MB/s | 200 kB 00:00 2023-07-20T02:40:43.475 INFO:teuthology.orchestra.run.smithi007.stdout:(73/131): python3-jwt-1.6.1-2.el8.noarch.rpm 464 kB/s | 43 kB 00:00 2023-07-20T02:40:43.609 INFO:teuthology.orchestra.run.smithi007.stdout:(74/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.4 MB/s | 220 kB 00:00 2023-07-20T02:40:43.635 INFO:teuthology.orchestra.run.smithi007.stdout:(75/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 303 kB/s | 48 kB 00:00 2023-07-20T02:40:43.743 INFO:teuthology.orchestra.run.smithi007.stdout:(76/131): python3-requests-oauthlib-1.0.0-1.el8 394 kB/s | 43 kB 00:00 2023-07-20T02:40:43.760 INFO:teuthology.orchestra.run.smithi007.stdout:(77/131): libsepol-devel-2.9-3.el8.x86_64.rpm 577 kB/s | 87 kB 00:00 2023-07-20T02:40:43.802 INFO:teuthology.orchestra.run.smithi007.stdout:(78/131): python3-pycparser-2.14-14.el8.noarch. 1.8 MB/s | 109 kB 00:00 2023-07-20T02:40:43.886 INFO:teuthology.orchestra.run.smithi007.stdout:(79/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.8 MB/s | 229 kB 00:00 2023-07-20T02:40:43.944 INFO:teuthology.orchestra.run.smithi007.stdout:(80/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 3.9 MB/s | 560 kB 00:00 2023-07-20T02:40:43.978 INFO:teuthology.orchestra.run.smithi007.stdout:(81/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 204 kB/s | 19 kB 00:00 2023-07-20T02:40:44.020 INFO:teuthology.orchestra.run.smithi007.stdout:(82/131): python3-oauthlib-2.1.0-1.el8.noarch.r 2.0 MB/s | 155 kB 00:00 2023-07-20T02:40:44.112 INFO:teuthology.orchestra.run.smithi007.stdout:(83/131): python3-cryptography-3.2.1-5.el8.x86_ 4.1 MB/s | 559 kB 00:00 2023-07-20T02:40:44.171 INFO:teuthology.orchestra.run.smithi007.stdout:(84/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 119 kB/s | 18 kB 00:00 2023-07-20T02:40:44.288 INFO:teuthology.orchestra.run.smithi007.stdout:(85/131): pcre2-devel-10.32-2.el8.x86_64.rpm 3.4 MB/s | 605 kB 00:00 2023-07-20T02:40:44.329 INFO:teuthology.orchestra.run.smithi007.stdout:(86/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.5 MB/s | 238 kB 00:00 2023-07-20T02:40:44.446 INFO:teuthology.orchestra.run.smithi007.stdout:(87/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 620 kB/s | 72 kB 00:00 2023-07-20T02:40:44.522 INFO:teuthology.orchestra.run.smithi007.stdout:(88/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 10 MB/s | 2.3 MB 00:00 2023-07-20T02:40:44.555 INFO:teuthology.orchestra.run.smithi007.stdout:(89/131): fuse-2.9.7-15.el8.x86_64.rpm 765 kB/s | 83 kB 00:00 2023-07-20T02:40:44.672 INFO:teuthology.orchestra.run.smithi007.stdout:(90/131): elfutils-libelf-devel-0.186-1.el8.x86 399 kB/s | 60 kB 00:00 2023-07-20T02:40:44.698 INFO:teuthology.orchestra.run.smithi007.stdout:(91/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 692 kB/s | 98 kB 00:00 2023-07-20T02:40:44.798 INFO:teuthology.orchestra.run.smithi007.stdout:(92/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 461 kB/s | 58 kB 00:00 2023-07-20T02:40:44.823 INFO:teuthology.orchestra.run.smithi007.stdout:(93/131): libcom_err-devel-1.45.6-4.el8.x86_64. 308 kB/s | 39 kB 00:00 2023-07-20T02:40:44.948 INFO:teuthology.orchestra.run.smithi007.stdout:(94/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 4.2 MB/s | 644 kB 00:00 2023-07-20T02:40:44.974 INFO:teuthology.orchestra.run.smithi007.stdout:(95/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 1.1 MB/s | 171 kB 00:00 2023-07-20T02:40:45.107 INFO:teuthology.orchestra.run.smithi007.stdout:(96/131): NetworkManager-initscripts-updown-1.3 868 kB/s | 138 kB 00:00 2023-07-20T02:40:45.241 INFO:teuthology.orchestra.run.smithi007.stdout:(97/131): python3-pyasn1-modules-0.3.7-6.el8.no 827 kB/s | 110 kB 00:00 2023-07-20T02:40:45.329 INFO:teuthology.orchestra.run.smithi007.stdout:(98/131): python3-pytz-2017.2-9.el8.noarch.rpm 617 kB/s | 54 kB 00:00 2023-07-20T02:40:45.508 INFO:teuthology.orchestra.run.smithi007.stdout:(99/131): python3-scipy-1.0.0-21.module+el8.5.0 26 MB/s | 14 MB 00:00 2023-07-20T02:40:45.542 INFO:teuthology.orchestra.run.smithi007.stdout:(100/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.2 MB/s | 259 kB 00:00 2023-07-20T02:40:45.575 INFO:teuthology.orchestra.run.smithi007.stdout:(101/131): socat-1.7.4.1-1.el8.x86_64.rpm 4.7 MB/s | 323 kB 00:00 2023-07-20T02:40:45.626 INFO:teuthology.orchestra.run.smithi007.stdout:(102/131): python3-markupsafe-0.23-19.el8.x86_6 463 kB/s | 39 kB 00:00 2023-07-20T02:40:45.718 INFO:teuthology.orchestra.run.smithi007.stdout:(103/131): python3-jinja2-2.10.1-3.el8.noarch.r 5.7 MB/s | 538 kB 00:00 2023-07-20T02:40:45.743 INFO:teuthology.orchestra.run.smithi007.stdout:(104/131): bison-3.0.4-10.el8.x86_64.rpm 4.0 MB/s | 688 kB 00:00 2023-07-20T02:40:45.844 INFO:teuthology.orchestra.run.smithi007.stdout:(105/131): python3-mako-1.0.6-13.el8.noarch.rpm 1.2 MB/s | 157 kB 00:00 2023-07-20T02:40:45.900 INFO:teuthology.orchestra.run.smithi007.stdout:(106/131): oniguruma-6.8.2-2.el8.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2023-07-20T02:40:45.986 INFO:teuthology.orchestra.run.smithi007.stdout:(107/131): automake-1.16.1-7.el8.noarch.rpm 4.9 MB/s | 713 kB 00:00 2023-07-20T02:40:46.053 INFO:teuthology.orchestra.run.smithi007.stdout:(108/131): python3-jsonpatch-1.21-2.el8.noarch. 399 kB/s | 27 kB 00:00 2023-07-20T02:40:46.145 INFO:teuthology.orchestra.run.smithi007.stdout:(109/131): python3-babel-2.5.1-7.el8.noarch.rpm 20 MB/s | 4.8 MB 00:00 2023-07-20T02:40:46.178 INFO:teuthology.orchestra.run.smithi007.stdout:(110/131): flex-2.6.1-9.el8.x86_64.rpm 2.5 MB/s | 320 kB 00:00 2023-07-20T02:40:46.329 INFO:teuthology.orchestra.run.smithi007.stdout:(111/131): python3-numpy-1.14.3-10.el8.x86_64.r 25 MB/s | 3.7 MB 00:00 2023-07-20T02:40:46.354 INFO:teuthology.orchestra.run.smithi007.stdout:(112/131): python3-prettytable-0.7.2-14.el8.noa 211 kB/s | 44 kB 00:00 2023-07-20T02:40:46.496 INFO:teuthology.orchestra.run.smithi007.stdout:(113/131): python3-werkzeug-0.12.2-4.el8.noarch 2.7 MB/s | 457 kB 00:00 2023-07-20T02:40:46.521 INFO:teuthology.orchestra.run.smithi007.stdout:(114/131): python3-numpy-f2py-1.14.3-10.el8.x86 1.3 MB/s | 225 kB 00:00 2023-07-20T02:40:46.555 INFO:teuthology.orchestra.run.smithi007.stdout:(115/131): uuid-1.6.2-43.el8.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2023-07-20T02:40:46.680 INFO:teuthology.orchestra.run.smithi007.stdout:(116/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 797 kB/s | 126 kB 00:00 2023-07-20T02:40:46.706 INFO:teuthology.orchestra.run.smithi007.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 133 kB/s | 20 kB 00:00 2023-07-20T02:40:46.823 INFO:teuthology.orchestra.run.smithi007.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 728 kB/s | 103 kB 00:00 2023-07-20T02:40:46.882 INFO:teuthology.orchestra.run.smithi007.stdout:(119/131): libtool-2.4.6-25.el8.x86_64.rpm 4.0 MB/s | 709 kB 00:00 2023-07-20T02:40:47.098 INFO:teuthology.orchestra.run.smithi007.stdout:(120/131): openblas-threads-0.3.15-3.el8.x86_64 18 MB/s | 4.9 MB 00:00 2023-07-20T02:40:47.123 INFO:teuthology.orchestra.run.smithi007.stdout:(121/131): libpq-13.5-1.el8.x86_64.rpm 818 kB/s | 198 kB 00:00 2023-07-20T02:40:47.257 INFO:teuthology.orchestra.run.smithi007.stdout:(122/131): jq-1.6-3.el8.x86_64.rpm 1.5 MB/s | 202 kB 00:00 2023-07-20T02:40:47.637 INFO:teuthology.orchestra.run.smithi007.stdout:(123/131): openblas-0.3.15-3.el8.x86_64.rpm 8.8 MB/s | 4.7 MB 00:00 2023-07-20T02:40:47.683 INFO:teuthology.orchestra.run.smithi007.stdout:(124/131): librdkafka-0.11.4-3.el8.x86_64.rpm 832 kB/s | 354 kB 00:00 2023-07-20T02:40:47.808 INFO:teuthology.orchestra.run.smithi007.stdout:(125/131): postgresql-contrib-10.21-2.module+el 6.3 MB/s | 808 kB 00:00 2023-07-20T02:40:47.933 INFO:teuthology.orchestra.run.smithi007.stdout:(126/131): postgresql-server-10.21-2.module+el8 17 MB/s | 5.1 MB 00:00 2023-07-20T02:40:47.992 INFO:teuthology.orchestra.run.smithi007.stdout:(127/131): postgresql-10.21-2.module+el8.6.0+15 8.2 MB/s | 1.5 MB 00:00 2023-07-20T02:40:48.459 INFO:teuthology.orchestra.run.smithi007.stdout:(128/131): thrift-0.13.0-2.el8.x86_64.rpm 323 kB/s | 1.7 MB 00:05 2023-07-20T02:40:48.576 INFO:teuthology.orchestra.run.smithi007.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 6.7 MB/s | 798 kB 00:00 2023-07-20T02:40:48.843 INFO:teuthology.orchestra.run.smithi007.stdout:(130/131): librados2-18.0.0-5040.g1b0b2eb0.el8. 4.1 MB/s | 3.7 MB 00:00 2023-07-20T02:40:48.901 INFO:teuthology.orchestra.run.smithi007.stdout:(131/131): librbd1-18.0.0-5040.g1b0b2eb0.el8.x8 4.0 MB/s | 3.6 MB 00:00 2023-07-20T02:40:48.906 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-07-20T02:40:48.907 INFO:teuthology.orchestra.run.smithi007.stdout:Total 12 MB/s | 206 MB 00:16 2023-07-20T02:40:50.595 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-07-20T02:40:50.742 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-07-20T02:40:50.742 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-07-20T02:40:52.341 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-07-20T02:40:52.346 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-07-20T02:40:57.467 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-07-20T02:40:57.931 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-20T02:40:58.170 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-20T02:40:58.340 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-20T02:40:58.352 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T02:40:58.353 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /sys 2023-07-20T02:40:58.353 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /proc 2023-07-20T02:40:58.353 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /dev 2023-07-20T02:40:58.353 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /run 2023-07-20T02:40:58.353 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /mnt 2023-07-20T02:40:58.353 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /var/tmp 2023-07-20T02:40:58.353 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /home 2023-07-20T02:40:58.354 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /tmp 2023-07-20T02:40:58.354 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /dev 2023-07-20T02:40:58.354 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-07-20T02:40:58.489 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-20T02:40:58.737 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T02:40:58.748 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-20T02:40:59.254 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-20T02:40:59.490 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 5/134 2023-07-20T02:40:59.976 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 5/134 2023-07-20T02:41:00.156 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T02:41:00.296 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T02:41:00.300 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T02:41:00.300 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /sys 2023-07-20T02:41:00.300 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /proc 2023-07-20T02:41:00.300 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /dev 2023-07-20T02:41:00.301 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /run 2023-07-20T02:41:00.301 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /mnt 2023-07-20T02:41:00.301 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /var/tmp 2023-07-20T02:41:00.301 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /home 2023-07-20T02:41:00.301 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /tmp 2023-07-20T02:41:00.301 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /dev 2023-07-20T02:41:00.301 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-07-20T02:41:00.323 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T02:41:00.498 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T02:41:00.533 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T02:41:00.605 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T02:41:00.627 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T02:41:00.752 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T02:41:00.779 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T02:41:00.989 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-07-20T02:41:01.010 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T02:41:01.254 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-20T02:41:01.464 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T02:41:01.504 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-20T02:41:01.669 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T02:41:01.686 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T02:41:01.940 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-20T02:41:02.199 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-20T02:41:02.203 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T02:41:02.355 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T02:41:02.356 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T02:41:02.488 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T02:41:02.572 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T02:41:02.664 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T02:41:02.743 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T02:41:03.100 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T02:41:03.195 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T02:41:03.364 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-07-20T02:41:03.385 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T02:41:03.402 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T02:41:03.519 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T02:41:03.542 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-07-20T02:41:03.727 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-07-20T02:41:03.744 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T02:41:03.866 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 102/134 2023-07-20T02:41:03.889 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-20T02:41:04.076 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-20T02:41:04.099 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 103/134 2023-07-20T02:41:04.264 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-20T02:41:04.362 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T02:41:04.574 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T02:41:04.816 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T02:41:05.048 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-20T02:41:05.214 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-20T02:41:05.260 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T02:41:05.448 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T02:41:05.582 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 102/134 2023-07-20T02:41:05.781 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T02:41:05.807 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 103/134 2023-07-20T02:41:05.943 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T02:41:05.943 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-20T02:41:05.968 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T02:41:06.034 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T02:41:06.147 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-07-20T02:41:06.183 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T02:41:06.453 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 107/134 2023-07-20T02:41:06.454 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T02:41:06.695 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T02:41:06.731 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T02:41:06.943 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T02:41:06.954 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T02:41:07.047 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-07-20T02:41:07.322 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-07-20T02:41:07.411 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T02:41:07.540 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-07-20T02:41:07.550 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T02:41:07.640 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T02:41:07.789 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-20T02:41:07.997 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 26/134 2023-07-20T02:41:08.081 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 107/134 2023-07-20T02:41:08.170 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 27/134 2023-07-20T02:41:08.274 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T02:41:08.308 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T02:41:08.320 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 110/134 2023-07-20T02:41:08.448 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 28/134 2023-07-20T02:41:08.483 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T02:41:08.495 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T02:41:08.502 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T02:41:08.536 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T02:41:08.622 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-20T02:41:08.684 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T02:41:08.727 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T02:41:08.885 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T02:41:09.237 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T02:41:09.393 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T02:41:09.786 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 110/134 2023-07-20T02:41:09.961 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T02:41:09.994 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T02:41:10.175 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T02:41:10.217 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T02:41:10.350 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T02:41:10.702 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T02:41:10.840 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-20T02:41:10.875 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T02:41:12.066 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-07-20T02:41:12.448 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-07-20T02:41:12.710 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-07-20T02:41:12.949 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-07-20T02:41:13.188 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-20T02:41:13.380 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-20T02:41:13.588 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 36/134 2023-07-20T02:41:13.780 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 37/134 2023-07-20T02:41:13.940 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-07-20T02:41:13.959 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T02:41:14.161 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-07-20T02:41:14.346 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-07-20T02:41:14.446 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-07-20T02:41:14.736 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-07-20T02:41:14.854 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 115/134 2023-07-20T02:41:14.954 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-07-20T02:41:15.113 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-20T02:41:15.156 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-07-20T02:41:15.216 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-20T02:41:15.328 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 117/134 2023-07-20T02:41:15.427 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T02:41:15.489 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-07-20T02:41:15.573 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-07-20T02:41:15.668 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-07-20T02:41:15.733 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-07-20T02:41:15.853 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-07-20T02:41:15.944 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-07-20T02:41:16.043 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-07-20T02:41:16.171 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-07-20T02:41:16.241 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-07-20T02:41:16.308 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 115/134 2023-07-20T02:41:16.329 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 121/134 2023-07-20T02:41:16.395 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-07-20T02:41:16.459 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-20T02:41:16.588 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-07-20T02:41:16.649 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-07-20T02:41:16.793 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 117/134 2023-07-20T02:41:16.874 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-07-20T02:41:17.040 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-07-20T02:41:17.240 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-07-20T02:41:17.354 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-07-20T02:41:17.457 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-07-20T02:41:17.671 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-07-20T02:41:17.703 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-07-20T02:41:17.844 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 121/134 2023-07-20T02:41:17.921 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-07-20T02:41:17.958 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-20T02:41:18.164 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-07-20T02:41:18.433 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-07-20T02:41:18.606 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T02:41:18.819 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-07-20T02:41:18.833 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T02:41:18.850 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T02:41:19.000 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 124/134 2023-07-20T02:41:19.048 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-07-20T02:41:19.257 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 125/134 2023-07-20T02:41:19.267 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-07-20T02:41:19.518 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-07-20T02:41:19.533 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 126/134 2023-07-20T02:41:19.697 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-07-20T02:41:19.868 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-20T02:41:19.920 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-07-20T02:41:20.032 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-07-20T02:41:20.116 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-07-20T02:41:20.172 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T02:41:20.282 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-07-20T02:41:20.300 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-07-20T02:41:20.313 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-07-20T02:41:20.432 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T02:41:20.450 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T02:41:20.458 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-20T02:41:20.466 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-20T02:41:20.573 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 124/134 2023-07-20T02:41:20.584 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-07-20T02:41:20.585 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T02:41:20.603 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-07-20T02:41:20.785 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T02:41:20.786 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T02:41:20.882 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-07-20T02:41:20.898 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 125/134 2023-07-20T02:41:21.033 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T02:41:21.141 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-07-20T02:41:21.185 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 126/134 2023-07-20T02:41:21.248 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T02:41:21.307 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-20T02:41:21.497 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-20T02:41:21.590 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-20T02:41:21.774 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T02:41:21.774 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 1/134 2023-07-20T02:41:21.774 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2/134 2023-07-20T02:41:21.775 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 3/134 2023-07-20T02:41:21.775 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 4/134 2023-07-20T02:41:21.775 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 5/134 2023-07-20T02:41:21.775 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T02:41:21.775 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T02:41:21.775 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T02:41:21.775 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 9/134 2023-07-20T02:41:21.775 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 10/134 2023-07-20T02:41:21.776 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 11/134 2023-07-20T02:41:21.776 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 12/134 2023-07-20T02:41:21.776 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 13/134 2023-07-20T02:41:21.776 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 14/134 2023-07-20T02:41:21.776 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 15/134 2023-07-20T02:41:21.776 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 16/134 2023-07-20T02:41:21.776 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 17/134 2023-07-20T02:41:21.777 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 18/134 2023-07-20T02:41:21.777 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 19/134 2023-07-20T02:41:21.777 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 20/134 2023-07-20T02:41:21.777 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 21/134 2023-07-20T02:41:21.777 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 22/134 2023-07-20T02:41:21.777 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 23/134 2023-07-20T02:41:21.777 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 24/134 2023-07-20T02:41:21.777 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 25/134 2023-07-20T02:41:21.778 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 26/134 2023-07-20T02:41:21.778 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 27/134 2023-07-20T02:41:21.778 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 28/134 2023-07-20T02:41:21.778 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 29/134 2023-07-20T02:41:21.778 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 30/134 2023-07-20T02:41:21.778 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 31/134 2023-07-20T02:41:21.785 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 32/134 2023-07-20T02:41:21.785 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 33/134 2023-07-20T02:41:21.785 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 34/134 2023-07-20T02:41:21.785 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 35/134 2023-07-20T02:41:21.785 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 36/134 2023-07-20T02:41:21.786 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 37/134 2023-07-20T02:41:21.786 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 38/134 2023-07-20T02:41:21.786 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-20T02:41:21.786 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-20T02:41:21.786 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-20T02:41:21.786 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-20T02:41:21.786 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-20T02:41:21.786 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-20T02:41:21.787 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-20T02:41:21.787 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-20T02:41:21.787 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-20T02:41:21.787 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-20T02:41:21.787 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-20T02:41:21.787 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-20T02:41:21.787 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-20T02:41:21.788 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-20T02:41:21.788 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-20T02:41:21.788 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-20T02:41:21.788 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-20T02:41:21.788 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-20T02:41:21.788 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-20T02:41:21.788 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-20T02:41:21.788 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-20T02:41:21.789 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-20T02:41:21.789 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-20T02:41:21.789 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-20T02:41:21.789 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-20T02:41:21.789 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-20T02:41:21.789 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-20T02:41:21.789 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-20T02:41:21.790 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-20T02:41:21.790 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-20T02:41:21.790 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-20T02:41:21.790 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-20T02:41:21.790 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-20T02:41:21.790 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-07-20T02:41:21.790 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-07-20T02:41:21.790 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-07-20T02:41:21.791 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-07-20T02:41:21.791 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-07-20T02:41:21.791 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-07-20T02:41:21.791 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-07-20T02:41:21.791 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-07-20T02:41:21.791 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-07-20T02:41:21.791 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-07-20T02:41:21.791 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-07-20T02:41:21.792 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-07-20T02:41:21.792 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-07-20T02:41:21.792 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-07-20T02:41:21.792 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-07-20T02:41:21.792 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-07-20T02:41:21.792 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-07-20T02:41:21.792 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-07-20T02:41:21.793 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-07-20T02:41:21.793 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-07-20T02:41:21.793 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-07-20T02:41:21.793 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-07-20T02:41:21.793 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-07-20T02:41:21.793 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-07-20T02:41:21.793 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-07-20T02:41:21.794 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-07-20T02:41:21.794 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-07-20T02:41:21.794 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-07-20T02:41:21.794 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-07-20T02:41:21.794 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-07-20T02:41:21.794 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-07-20T02:41:21.794 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-07-20T02:41:21.794 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-07-20T02:41:21.795 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-07-20T02:41:21.795 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-07-20T02:41:21.795 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-07-20T02:41:21.795 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-07-20T02:41:21.795 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-07-20T02:41:21.795 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-07-20T02:41:21.795 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-07-20T02:41:21.795 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-07-20T02:41:21.796 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-07-20T02:41:21.796 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-07-20T02:41:21.796 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-07-20T02:41:21.796 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-07-20T02:41:21.796 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-07-20T02:41:21.796 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-07-20T02:41:21.796 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-07-20T02:41:21.797 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-07-20T02:41:21.797 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-07-20T02:41:21.797 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-07-20T02:41:21.797 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-07-20T02:41:21.797 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-07-20T02:41:21.797 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-07-20T02:41:21.797 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-07-20T02:41:21.797 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-07-20T02:41:21.798 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-07-20T02:41:21.798 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 129/134 2023-07-20T02:41:21.798 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-20T02:41:21.798 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 131/134 2023-07-20T02:41:21.798 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T02:41:21.798 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-20T02:41:21.799 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-07-20T02:41:21.812 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-20T02:41:21.869 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-07-20T02:41:22.038 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-20T02:41:22.056 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-07-20T02:41:22.068 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-07-20T02:41:22.200 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-20T02:41:22.256 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-20T02:41:22.264 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-20T02:41:22.391 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-07-20T02:41:22.392 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T02:41:22.401 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-20T02:41:22.576 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T02:41:22.576 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T02:41:22.622 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-20T02:41:22.837 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T02:41:22.848 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-20T02:41:23.030 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T02:41:23.088 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-20T02:41:23.341 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-20T02:41:23.503 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-20T02:41:23.572 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T02:41:23.583 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 1/134 2023-07-20T02:41:23.583 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2/134 2023-07-20T02:41:23.583 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 3/134 2023-07-20T02:41:23.583 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 4/134 2023-07-20T02:41:23.583 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 5/134 2023-07-20T02:41:23.583 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T02:41:23.584 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T02:41:23.584 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T02:41:23.584 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 9/134 2023-07-20T02:41:23.584 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 10/134 2023-07-20T02:41:23.584 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 11/134 2023-07-20T02:41:23.584 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 12/134 2023-07-20T02:41:23.584 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 13/134 2023-07-20T02:41:23.585 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 14/134 2023-07-20T02:41:23.585 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 15/134 2023-07-20T02:41:23.585 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 16/134 2023-07-20T02:41:23.585 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 17/134 2023-07-20T02:41:23.586 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 18/134 2023-07-20T02:41:23.586 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 19/134 2023-07-20T02:41:23.586 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 20/134 2023-07-20T02:41:23.586 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 21/134 2023-07-20T02:41:23.587 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 22/134 2023-07-20T02:41:23.587 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 23/134 2023-07-20T02:41:23.587 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 24/134 2023-07-20T02:41:23.588 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 25/134 2023-07-20T02:41:23.588 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 26/134 2023-07-20T02:41:23.588 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 27/134 2023-07-20T02:41:23.589 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 28/134 2023-07-20T02:41:23.589 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 29/134 2023-07-20T02:41:23.589 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 30/134 2023-07-20T02:41:23.589 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 31/134 2023-07-20T02:41:23.590 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 32/134 2023-07-20T02:41:23.590 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 33/134 2023-07-20T02:41:23.590 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 34/134 2023-07-20T02:41:23.591 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 35/134 2023-07-20T02:41:23.591 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 36/134 2023-07-20T02:41:23.591 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 37/134 2023-07-20T02:41:23.592 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 38/134 2023-07-20T02:41:23.592 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-20T02:41:23.592 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-20T02:41:23.593 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-20T02:41:23.593 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-20T02:41:23.593 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-20T02:41:23.593 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-20T02:41:23.594 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-20T02:41:23.594 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-20T02:41:23.594 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-20T02:41:23.594 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-20T02:41:23.594 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-20T02:41:23.595 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-20T02:41:23.595 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-20T02:41:23.595 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-20T02:41:23.595 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-20T02:41:23.596 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-20T02:41:23.596 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-20T02:41:23.596 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-20T02:41:23.597 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-20T02:41:23.597 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-20T02:41:23.597 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-20T02:41:23.597 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-20T02:41:23.597 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-20T02:41:23.597 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-20T02:41:23.597 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-20T02:41:23.598 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-20T02:41:23.598 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-20T02:41:23.598 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-20T02:41:23.598 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-20T02:41:23.598 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-20T02:41:23.598 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-20T02:41:23.598 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-20T02:41:23.599 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-20T02:41:23.599 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-07-20T02:41:23.599 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-07-20T02:41:23.599 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-07-20T02:41:23.599 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-07-20T02:41:23.599 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-07-20T02:41:23.599 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-07-20T02:41:23.599 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-07-20T02:41:23.600 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-07-20T02:41:23.600 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-07-20T02:41:23.600 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-07-20T02:41:23.600 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-07-20T02:41:23.600 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-07-20T02:41:23.601 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-07-20T02:41:23.601 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-07-20T02:41:23.601 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-07-20T02:41:23.601 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-07-20T02:41:23.601 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-07-20T02:41:23.601 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-07-20T02:41:23.601 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-07-20T02:41:23.601 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-07-20T02:41:23.602 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-07-20T02:41:23.602 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-07-20T02:41:23.602 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-07-20T02:41:23.602 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-07-20T02:41:23.602 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-07-20T02:41:23.602 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-07-20T02:41:23.602 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-07-20T02:41:23.603 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-07-20T02:41:23.603 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-07-20T02:41:23.603 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-07-20T02:41:23.603 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-07-20T02:41:23.603 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-07-20T02:41:23.603 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-07-20T02:41:23.603 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-07-20T02:41:23.603 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-07-20T02:41:23.604 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-07-20T02:41:23.604 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-07-20T02:41:23.604 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-07-20T02:41:23.604 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-07-20T02:41:23.604 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-07-20T02:41:23.604 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-07-20T02:41:23.604 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-07-20T02:41:23.605 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-07-20T02:41:23.605 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-07-20T02:41:23.605 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-07-20T02:41:23.605 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-07-20T02:41:23.605 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-07-20T02:41:23.605 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-07-20T02:41:23.605 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-07-20T02:41:23.605 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-07-20T02:41:23.606 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-07-20T02:41:23.606 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-07-20T02:41:23.606 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-07-20T02:41:23.606 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-07-20T02:41:23.607 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-07-20T02:41:23.607 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-07-20T02:41:23.607 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-07-20T02:41:23.607 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 129/134 2023-07-20T02:41:23.607 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-20T02:41:23.607 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 131/134 2023-07-20T02:41:23.607 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T02:41:23.608 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-20T02:41:23.786 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-20T02:41:24.045 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-20T02:41:24.234 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T02:41:24.234 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:07 ago on Thu 20 Jul 2023 02:39:17 AM UTC. 2023-07-20T02:41:24.286 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-20T02:41:24.570 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-20T02:41:24.838 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-20T02:41:25.024 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-20T02:41:25.262 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-20T02:41:26.144 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T02:41:26.144 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:09 ago on Thu 20 Jul 2023 02:39:17 AM UTC. 2023-07-20T02:41:26.740 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-20T02:41:27.043 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-20T02:41:27.319 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-20T02:41:27.690 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-20T02:41:28.001 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-20T02:41:28.277 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-20T02:41:29.271 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-20T02:41:29.502 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T02:41:29.599 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T02:41:29.786 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 93/134 2023-07-20T02:41:34.184 INFO:teuthology.orchestra.run.smithi188.stdout:Installed products updated. 2023-07-20T02:41:34.396 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-07-20T02:41:34.397 INFO:teuthology.orchestra.run.smithi188.stdout:Upgraded: 2023-07-20T02:41:34.397 INFO:teuthology.orchestra.run.smithi188.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-20T02:41:34.397 INFO:teuthology.orchestra.run.smithi188.stdout: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.397 INFO:teuthology.orchestra.run.smithi188.stdout: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.398 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2023-07-20T02:41:34.398 INFO:teuthology.orchestra.run.smithi188.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-20T02:41:34.398 INFO:teuthology.orchestra.run.smithi188.stdout: automake-1.16.1-7.el8.noarch 2023-07-20T02:41:34.398 INFO:teuthology.orchestra.run.smithi188.stdout: bison-3.0.4-10.el8.x86_64 2023-07-20T02:41:34.398 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.399 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.399 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.399 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.399 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:41:34.400 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.400 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.400 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.401 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:41:34.401 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:41:34.401 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:41:34.401 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:41:34.402 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:41:34.402 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.402 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.402 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:41:34.403 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.403 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.403 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.403 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:41:34.404 INFO:teuthology.orchestra.run.smithi188.stdout: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:41:34.404 INFO:teuthology.orchestra.run.smithi188.stdout: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.404 INFO:teuthology.orchestra.run.smithi188.stdout: cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:41:34.404 INFO:teuthology.orchestra.run.smithi188.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-20T02:41:34.405 INFO:teuthology.orchestra.run.smithi188.stdout: flex-2.6.1-9.el8.x86_64 2023-07-20T02:41:34.405 INFO:teuthology.orchestra.run.smithi188.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-20T02:41:34.405 INFO:teuthology.orchestra.run.smithi188.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-20T02:41:34.405 INFO:teuthology.orchestra.run.smithi188.stdout: jq-1.6-3.el8.x86_64 2023-07-20T02:41:34.405 INFO:teuthology.orchestra.run.smithi188.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-20T02:41:34.405 INFO:teuthology.orchestra.run.smithi188.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-20T02:41:34.405 INFO:teuthology.orchestra.run.smithi188.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-20T02:41:34.406 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.406 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.407 INFO:teuthology.orchestra.run.smithi188.stdout: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.407 INFO:teuthology.orchestra.run.smithi188.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-20T02:41:34.407 INFO:teuthology.orchestra.run.smithi188.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-20T02:41:34.407 INFO:teuthology.orchestra.run.smithi188.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-20T02:41:34.407 INFO:teuthology.orchestra.run.smithi188.stdout: libpq-13.5-1.el8.x86_64 2023-07-20T02:41:34.408 INFO:teuthology.orchestra.run.smithi188.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-20T02:41:34.408 INFO:teuthology.orchestra.run.smithi188.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-20T02:41:34.408 INFO:teuthology.orchestra.run.smithi188.stdout: librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.408 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.408 INFO:teuthology.orchestra.run.smithi188.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-20T02:41:34.408 INFO:teuthology.orchestra.run.smithi188.stdout: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.409 INFO:teuthology.orchestra.run.smithi188.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-20T02:41:34.409 INFO:teuthology.orchestra.run.smithi188.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-20T02:41:34.409 INFO:teuthology.orchestra.run.smithi188.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-20T02:41:34.409 INFO:teuthology.orchestra.run.smithi188.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-20T02:41:34.409 INFO:teuthology.orchestra.run.smithi188.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-20T02:41:34.410 INFO:teuthology.orchestra.run.smithi188.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-20T02:41:34.410 INFO:teuthology.orchestra.run.smithi188.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-20T02:41:34.410 INFO:teuthology.orchestra.run.smithi188.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-20T02:41:34.410 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-20T02:41:34.410 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-20T02:41:34.410 INFO:teuthology.orchestra.run.smithi188.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-20T02:41:34.410 INFO:teuthology.orchestra.run.smithi188.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-20T02:41:34.411 INFO:teuthology.orchestra.run.smithi188.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-20T02:41:34.411 INFO:teuthology.orchestra.run.smithi188.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-20T02:41:34.411 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T02:41:34.411 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T02:41:34.411 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T02:41:34.411 INFO:teuthology.orchestra.run.smithi188.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-20T02:41:34.412 INFO:teuthology.orchestra.run.smithi188.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-20T02:41:34.412 INFO:teuthology.orchestra.run.smithi188.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-20T02:41:34.412 INFO:teuthology.orchestra.run.smithi188.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-20T02:41:34.412 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-20T02:41:34.412 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.412 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.412 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.413 INFO:teuthology.orchestra.run.smithi188.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-20T02:41:34.413 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-20T02:41:34.413 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-20T02:41:34.413 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-20T02:41:34.413 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-20T02:41:34.413 INFO:teuthology.orchestra.run.smithi188.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-20T02:41:34.413 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-20T02:41:34.413 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-20T02:41:34.414 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-20T02:41:34.414 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-20T02:41:34.414 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-20T02:41:34.414 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-20T02:41:34.414 INFO:teuthology.orchestra.run.smithi188.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-20T02:41:34.414 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-20T02:41:34.414 INFO:teuthology.orchestra.run.smithi188.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-20T02:41:34.414 INFO:teuthology.orchestra.run.smithi188.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-20T02:41:34.415 INFO:teuthology.orchestra.run.smithi188.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-20T02:41:34.415 INFO:teuthology.orchestra.run.smithi188.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-20T02:41:34.415 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-20T02:41:34.415 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-20T02:41:34.415 INFO:teuthology.orchestra.run.smithi188.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-20T02:41:34.415 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-20T02:41:34.415 INFO:teuthology.orchestra.run.smithi188.stdout: python3-portend-2.6-1.el8.noarch 2023-07-20T02:41:34.416 INFO:teuthology.orchestra.run.smithi188.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-20T02:41:34.416 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-20T02:41:34.416 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-20T02:41:34.416 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-20T02:41:34.416 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-20T02:41:34.416 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-20T02:41:34.416 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.417 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.417 INFO:teuthology.orchestra.run.smithi188.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-20T02:41:34.417 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-20T02:41:34.417 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.417 INFO:teuthology.orchestra.run.smithi188.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-20T02:41:34.417 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-20T02:41:34.417 INFO:teuthology.orchestra.run.smithi188.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-20T02:41:34.418 INFO:teuthology.orchestra.run.smithi188.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-20T02:41:34.418 INFO:teuthology.orchestra.run.smithi188.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-20T02:41:34.418 INFO:teuthology.orchestra.run.smithi188.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-20T02:41:34.418 INFO:teuthology.orchestra.run.smithi188.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-20T02:41:34.418 INFO:teuthology.orchestra.run.smithi188.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-20T02:41:34.418 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-20T02:41:34.418 INFO:teuthology.orchestra.run.smithi188.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-20T02:41:34.419 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-20T02:41:34.419 INFO:teuthology.orchestra.run.smithi188.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-20T02:41:34.419 INFO:teuthology.orchestra.run.smithi188.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-20T02:41:34.419 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.419 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.419 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.419 INFO:teuthology.orchestra.run.smithi188.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-20T02:41:34.419 INFO:teuthology.orchestra.run.smithi188.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-20T02:41:34.420 INFO:teuthology.orchestra.run.smithi188.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-20T02:41:34.420 INFO:teuthology.orchestra.run.smithi188.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-20T02:41:34.420 INFO:teuthology.orchestra.run.smithi188.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-20T02:41:34.420 INFO:teuthology.orchestra.run.smithi188.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-20T02:41:34.421 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-07-20T02:41:34.421 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2023-07-20T02:41:34.492 INFO:teuthology.orchestra.run.smithi158.stdout:Installed products updated. 2023-07-20T02:41:34.605 DEBUG:teuthology.parallel:result is None 2023-07-20T02:41:34.706 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-07-20T02:41:34.706 INFO:teuthology.orchestra.run.smithi158.stdout:Upgraded: 2023-07-20T02:41:34.707 INFO:teuthology.orchestra.run.smithi158.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-20T02:41:34.707 INFO:teuthology.orchestra.run.smithi158.stdout: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.707 INFO:teuthology.orchestra.run.smithi158.stdout: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.707 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2023-07-20T02:41:34.707 INFO:teuthology.orchestra.run.smithi158.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-20T02:41:34.707 INFO:teuthology.orchestra.run.smithi158.stdout: automake-1.16.1-7.el8.noarch 2023-07-20T02:41:34.708 INFO:teuthology.orchestra.run.smithi158.stdout: bison-3.0.4-10.el8.x86_64 2023-07-20T02:41:34.708 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.708 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.709 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.709 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.709 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:41:34.709 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.710 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.710 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.710 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:41:34.710 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:41:34.710 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:41:34.710 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:41:34.710 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:41:34.711 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.711 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.711 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:41:34.711 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.711 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.711 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.711 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:41:34.712 INFO:teuthology.orchestra.run.smithi158.stdout: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:41:34.712 INFO:teuthology.orchestra.run.smithi158.stdout: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.712 INFO:teuthology.orchestra.run.smithi158.stdout: cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:41:34.712 INFO:teuthology.orchestra.run.smithi158.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-20T02:41:34.712 INFO:teuthology.orchestra.run.smithi158.stdout: flex-2.6.1-9.el8.x86_64 2023-07-20T02:41:34.712 INFO:teuthology.orchestra.run.smithi158.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-20T02:41:34.712 INFO:teuthology.orchestra.run.smithi158.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-20T02:41:34.712 INFO:teuthology.orchestra.run.smithi158.stdout: jq-1.6-3.el8.x86_64 2023-07-20T02:41:34.713 INFO:teuthology.orchestra.run.smithi158.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-20T02:41:34.713 INFO:teuthology.orchestra.run.smithi158.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-20T02:41:34.713 INFO:teuthology.orchestra.run.smithi158.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-20T02:41:34.713 INFO:teuthology.orchestra.run.smithi158.stdout: libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.713 INFO:teuthology.orchestra.run.smithi158.stdout: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.713 INFO:teuthology.orchestra.run.smithi158.stdout: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.713 INFO:teuthology.orchestra.run.smithi158.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-20T02:41:34.713 INFO:teuthology.orchestra.run.smithi158.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-20T02:41:34.714 INFO:teuthology.orchestra.run.smithi158.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-20T02:41:34.714 INFO:teuthology.orchestra.run.smithi158.stdout: libpq-13.5-1.el8.x86_64 2023-07-20T02:41:34.714 INFO:teuthology.orchestra.run.smithi158.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-20T02:41:34.714 INFO:teuthology.orchestra.run.smithi158.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-20T02:41:34.714 INFO:teuthology.orchestra.run.smithi158.stdout: librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.714 INFO:teuthology.orchestra.run.smithi158.stdout: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.714 INFO:teuthology.orchestra.run.smithi158.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-20T02:41:34.715 INFO:teuthology.orchestra.run.smithi158.stdout: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.715 INFO:teuthology.orchestra.run.smithi158.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-20T02:41:34.715 INFO:teuthology.orchestra.run.smithi158.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-20T02:41:34.715 INFO:teuthology.orchestra.run.smithi158.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-20T02:41:34.715 INFO:teuthology.orchestra.run.smithi158.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-20T02:41:34.715 INFO:teuthology.orchestra.run.smithi158.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-20T02:41:34.715 INFO:teuthology.orchestra.run.smithi158.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-20T02:41:34.715 INFO:teuthology.orchestra.run.smithi158.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-20T02:41:34.716 INFO:teuthology.orchestra.run.smithi158.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-20T02:41:34.716 INFO:teuthology.orchestra.run.smithi158.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-20T02:41:34.716 INFO:teuthology.orchestra.run.smithi158.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-20T02:41:34.716 INFO:teuthology.orchestra.run.smithi158.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-20T02:41:34.716 INFO:teuthology.orchestra.run.smithi158.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-20T02:41:34.716 INFO:teuthology.orchestra.run.smithi158.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-20T02:41:34.716 INFO:teuthology.orchestra.run.smithi158.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-20T02:41:34.717 INFO:teuthology.orchestra.run.smithi158.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T02:41:34.717 INFO:teuthology.orchestra.run.smithi158.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T02:41:34.717 INFO:teuthology.orchestra.run.smithi158.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T02:41:34.717 INFO:teuthology.orchestra.run.smithi158.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-20T02:41:34.717 INFO:teuthology.orchestra.run.smithi158.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-20T02:41:34.717 INFO:teuthology.orchestra.run.smithi158.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-20T02:41:34.718 INFO:teuthology.orchestra.run.smithi158.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-20T02:41:34.718 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-20T02:41:34.718 INFO:teuthology.orchestra.run.smithi158.stdout: python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.718 INFO:teuthology.orchestra.run.smithi158.stdout: python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.719 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.719 INFO:teuthology.orchestra.run.smithi158.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-20T02:41:34.719 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-20T02:41:34.719 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-20T02:41:34.719 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-20T02:41:34.719 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-20T02:41:34.719 INFO:teuthology.orchestra.run.smithi158.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-20T02:41:34.720 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-20T02:41:34.720 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-20T02:41:34.720 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-20T02:41:34.720 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-20T02:41:34.720 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-20T02:41:34.720 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-20T02:41:34.720 INFO:teuthology.orchestra.run.smithi158.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-20T02:41:34.720 INFO:teuthology.orchestra.run.smithi158.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-20T02:41:34.721 INFO:teuthology.orchestra.run.smithi158.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-20T02:41:34.721 INFO:teuthology.orchestra.run.smithi158.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-20T02:41:34.721 INFO:teuthology.orchestra.run.smithi158.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-20T02:41:34.721 INFO:teuthology.orchestra.run.smithi158.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-20T02:41:34.721 INFO:teuthology.orchestra.run.smithi158.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-20T02:41:34.721 INFO:teuthology.orchestra.run.smithi158.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-20T02:41:34.721 INFO:teuthology.orchestra.run.smithi158.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-20T02:41:34.722 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-20T02:41:34.722 INFO:teuthology.orchestra.run.smithi158.stdout: python3-portend-2.6-1.el8.noarch 2023-07-20T02:41:34.722 INFO:teuthology.orchestra.run.smithi158.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-20T02:41:34.722 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-20T02:41:34.722 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-20T02:41:34.722 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-20T02:41:34.722 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-20T02:41:34.722 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-20T02:41:34.723 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.723 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.723 INFO:teuthology.orchestra.run.smithi158.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-20T02:41:34.723 INFO:teuthology.orchestra.run.smithi158.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-20T02:41:34.723 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.723 INFO:teuthology.orchestra.run.smithi158.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-20T02:41:34.723 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-20T02:41:34.724 INFO:teuthology.orchestra.run.smithi158.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-20T02:41:34.724 INFO:teuthology.orchestra.run.smithi158.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-20T02:41:34.724 INFO:teuthology.orchestra.run.smithi158.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-20T02:41:34.724 INFO:teuthology.orchestra.run.smithi158.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-20T02:41:34.724 INFO:teuthology.orchestra.run.smithi158.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-20T02:41:34.724 INFO:teuthology.orchestra.run.smithi158.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-20T02:41:34.724 INFO:teuthology.orchestra.run.smithi158.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-20T02:41:34.724 INFO:teuthology.orchestra.run.smithi158.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-20T02:41:34.725 INFO:teuthology.orchestra.run.smithi158.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-20T02:41:34.725 INFO:teuthology.orchestra.run.smithi158.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-20T02:41:34.725 INFO:teuthology.orchestra.run.smithi158.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-20T02:41:34.725 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.725 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.725 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:41:34.725 INFO:teuthology.orchestra.run.smithi158.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-20T02:41:34.725 INFO:teuthology.orchestra.run.smithi158.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-20T02:41:34.726 INFO:teuthology.orchestra.run.smithi158.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-20T02:41:34.726 INFO:teuthology.orchestra.run.smithi158.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-20T02:41:34.726 INFO:teuthology.orchestra.run.smithi158.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-20T02:41:34.726 INFO:teuthology.orchestra.run.smithi158.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-20T02:41:34.727 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-07-20T02:41:34.727 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2023-07-20T02:41:34.892 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T02:41:34.905 DEBUG:teuthology.parallel:result is None 2023-07-20T02:41:35.244 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T02:41:35.983 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T02:41:36.401 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T02:41:36.536 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T02:41:36.685 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T02:41:58.358 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T02:41:58.358 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /sys 2023-07-20T02:41:58.358 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /proc 2023-07-20T02:41:58.359 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /dev 2023-07-20T02:41:58.359 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /run 2023-07-20T02:41:58.359 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /mnt 2023-07-20T02:41:58.359 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /var/tmp 2023-07-20T02:41:58.359 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /home 2023-07-20T02:41:58.359 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /tmp 2023-07-20T02:41:58.359 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /dev 2023-07-20T02:41:58.360 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:41:58.785 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T02:42:00.643 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T02:42:00.847 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T02:42:01.131 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T02:42:01.306 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T02:42:01.865 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T02:42:02.089 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T02:42:02.133 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T02:42:04.911 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T02:42:05.117 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T02:42:05.256 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 102/134 2023-07-20T02:42:05.504 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 103/134 2023-07-20T02:42:05.688 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T02:42:05.965 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T02:42:06.213 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T02:42:07.402 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T02:42:07.571 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T02:42:07.709 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T02:42:08.141 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 107/134 2023-07-20T02:42:08.395 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T02:42:08.463 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T02:42:08.646 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T02:42:08.675 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T02:42:10.189 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 110/134 2023-07-20T02:42:10.388 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T02:42:10.453 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T02:42:10.588 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T02:42:10.667 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T02:42:10.847 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T02:42:11.288 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T02:42:11.447 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T02:42:16.790 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T02:42:18.173 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 115/134 2023-07-20T02:42:18.505 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-07-20T02:42:18.685 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 117/134 2023-07-20T02:42:18.974 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-07-20T02:42:19.153 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-07-20T02:42:19.440 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-07-20T02:42:19.692 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-07-20T02:42:19.934 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 121/134 2023-07-20T02:42:20.088 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-20T02:42:23.553 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T02:42:23.803 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T02:42:23.842 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T02:42:24.129 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 124/134 2023-07-20T02:42:24.347 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 125/134 2023-07-20T02:42:24.721 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 126/134 2023-07-20T02:42:25.059 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-20T02:42:25.285 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-07-20T02:42:25.402 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-07-20T02:42:25.611 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-07-20T02:42:25.639 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-07-20T02:42:25.894 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-20T02:42:25.917 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-20T02:42:26.089 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-07-20T02:42:26.090 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T02:42:26.351 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T02:42:26.351 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T02:42:26.643 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T02:42:26.911 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T02:42:27.192 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T02:42:27.192 INFO:teuthology.orchestra.run.smithi007.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2023-07-20T02:42:27.193 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:42:27.818 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 1/134 2023-07-20T02:42:27.818 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2/134 2023-07-20T02:42:27.818 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 3/134 2023-07-20T02:42:27.818 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 4/134 2023-07-20T02:42:27.818 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 5/134 2023-07-20T02:42:27.819 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T02:42:27.819 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T02:42:27.819 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T02:42:27.819 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 9/134 2023-07-20T02:42:27.819 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 10/134 2023-07-20T02:42:27.819 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 11/134 2023-07-20T02:42:27.819 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 12/134 2023-07-20T02:42:27.820 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 13/134 2023-07-20T02:42:27.820 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 14/134 2023-07-20T02:42:27.820 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 15/134 2023-07-20T02:42:27.820 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 16/134 2023-07-20T02:42:27.820 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 17/134 2023-07-20T02:42:27.820 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 18/134 2023-07-20T02:42:27.820 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 19/134 2023-07-20T02:42:27.821 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 20/134 2023-07-20T02:42:27.821 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 21/134 2023-07-20T02:42:27.821 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 22/134 2023-07-20T02:42:27.821 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 23/134 2023-07-20T02:42:27.821 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 24/134 2023-07-20T02:42:27.821 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 25/134 2023-07-20T02:42:27.821 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 26/134 2023-07-20T02:42:27.821 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 27/134 2023-07-20T02:42:27.822 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 28/134 2023-07-20T02:42:27.822 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 29/134 2023-07-20T02:42:27.822 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 30/134 2023-07-20T02:42:27.829 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 31/134 2023-07-20T02:42:27.829 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 32/134 2023-07-20T02:42:27.829 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 33/134 2023-07-20T02:42:27.829 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 34/134 2023-07-20T02:42:27.829 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 35/134 2023-07-20T02:42:27.830 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 36/134 2023-07-20T02:42:27.830 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 37/134 2023-07-20T02:42:27.830 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 38/134 2023-07-20T02:42:27.830 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-20T02:42:27.830 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-20T02:42:27.830 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-20T02:42:27.830 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-20T02:42:27.830 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-20T02:42:27.831 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-20T02:42:27.831 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-20T02:42:27.831 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-20T02:42:27.831 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-20T02:42:27.831 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-20T02:42:27.831 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-20T02:42:27.831 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-20T02:42:27.832 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-20T02:42:27.832 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-20T02:42:27.832 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-20T02:42:27.832 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-20T02:42:27.832 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-20T02:42:27.832 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-20T02:42:27.833 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-20T02:42:27.833 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-20T02:42:27.833 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-20T02:42:27.833 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-20T02:42:27.833 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-20T02:42:27.833 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-20T02:42:27.833 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-20T02:42:27.833 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-20T02:42:27.834 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-20T02:42:27.834 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-20T02:42:27.834 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-20T02:42:27.834 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-20T02:42:27.834 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-20T02:42:27.834 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-20T02:42:27.834 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-20T02:42:27.835 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-07-20T02:42:27.835 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-07-20T02:42:27.835 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-07-20T02:42:27.835 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-07-20T02:42:27.835 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-07-20T02:42:27.835 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-07-20T02:42:27.835 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-07-20T02:42:27.835 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-07-20T02:42:27.836 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-07-20T02:42:27.836 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-07-20T02:42:27.836 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-07-20T02:42:27.836 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-07-20T02:42:27.836 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-07-20T02:42:27.836 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-07-20T02:42:27.836 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-07-20T02:42:27.836 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-07-20T02:42:27.837 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-07-20T02:42:27.837 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-07-20T02:42:27.837 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-07-20T02:42:27.837 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-07-20T02:42:27.837 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-07-20T02:42:27.837 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-07-20T02:42:27.837 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-07-20T02:42:27.838 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-07-20T02:42:27.838 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-07-20T02:42:27.838 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-07-20T02:42:27.838 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-07-20T02:42:27.838 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-07-20T02:42:27.838 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-07-20T02:42:27.838 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-07-20T02:42:27.838 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-07-20T02:42:27.839 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-07-20T02:42:27.839 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-07-20T02:42:27.839 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-07-20T02:42:27.839 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-07-20T02:42:27.839 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-07-20T02:42:27.839 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-07-20T02:42:27.839 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-07-20T02:42:27.839 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-07-20T02:42:27.840 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-07-20T02:42:27.840 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-07-20T02:42:27.840 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-07-20T02:42:27.840 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-07-20T02:42:27.840 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-07-20T02:42:27.840 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-07-20T02:42:27.840 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-07-20T02:42:27.841 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-07-20T02:42:27.841 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-07-20T02:42:27.841 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-07-20T02:42:27.841 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-07-20T02:42:27.841 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-07-20T02:42:27.841 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-07-20T02:42:27.841 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-07-20T02:42:27.841 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-07-20T02:42:27.842 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-07-20T02:42:27.842 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-07-20T02:42:27.842 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-07-20T02:42:27.842 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 129/134 2023-07-20T02:42:27.842 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-20T02:42:27.842 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 131/134 2023-07-20T02:42:27.842 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T02:42:27.842 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-20T02:42:30.569 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T02:42:30.569 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:11 ago on Thu 20 Jul 2023 02:40:19 AM UTC. 2023-07-20T02:42:41.675 INFO:teuthology.orchestra.run.smithi007.stdout:Installed products updated. 2023-07-20T02:42:41.916 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:42:41.917 INFO:teuthology.orchestra.run.smithi007.stdout:Upgraded: 2023-07-20T02:42:41.917 INFO:teuthology.orchestra.run.smithi007.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-20T02:42:41.917 INFO:teuthology.orchestra.run.smithi007.stdout: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:42:41.917 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:42:41.917 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-07-20T02:42:41.917 INFO:teuthology.orchestra.run.smithi007.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-20T02:42:41.917 INFO:teuthology.orchestra.run.smithi007.stdout: automake-1.16.1-7.el8.noarch 2023-07-20T02:42:41.918 INFO:teuthology.orchestra.run.smithi007.stdout: bison-3.0.4-10.el8.x86_64 2023-07-20T02:42:41.918 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:42:41.918 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:42:41.918 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:42:41.918 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:42:41.918 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:42:41.918 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:42:41.919 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:42:41.919 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:42:41.919 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:42:41.919 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:42:41.919 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:42:41.919 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:42:41.919 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:42:41.919 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:42:41.920 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:42:41.920 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:42:41.920 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:42:41.920 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:42:41.920 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:42:41.920 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:42:41.920 INFO:teuthology.orchestra.run.smithi007.stdout: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:42:41.921 INFO:teuthology.orchestra.run.smithi007.stdout: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:42:41.921 INFO:teuthology.orchestra.run.smithi007.stdout: cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T02:42:41.921 INFO:teuthology.orchestra.run.smithi007.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-20T02:42:41.921 INFO:teuthology.orchestra.run.smithi007.stdout: flex-2.6.1-9.el8.x86_64 2023-07-20T02:42:41.921 INFO:teuthology.orchestra.run.smithi007.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-20T02:42:41.921 INFO:teuthology.orchestra.run.smithi007.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-20T02:42:41.921 INFO:teuthology.orchestra.run.smithi007.stdout: jq-1.6-3.el8.x86_64 2023-07-20T02:42:41.921 INFO:teuthology.orchestra.run.smithi007.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-20T02:42:41.922 INFO:teuthology.orchestra.run.smithi007.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-20T02:42:41.922 INFO:teuthology.orchestra.run.smithi007.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-20T02:42:41.922 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:42:41.922 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:42:41.922 INFO:teuthology.orchestra.run.smithi007.stdout: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:42:41.922 INFO:teuthology.orchestra.run.smithi007.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-20T02:42:41.922 INFO:teuthology.orchestra.run.smithi007.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-20T02:42:41.922 INFO:teuthology.orchestra.run.smithi007.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-20T02:42:41.923 INFO:teuthology.orchestra.run.smithi007.stdout: libpq-13.5-1.el8.x86_64 2023-07-20T02:42:41.923 INFO:teuthology.orchestra.run.smithi007.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-20T02:42:41.923 INFO:teuthology.orchestra.run.smithi007.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-20T02:42:41.923 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:42:41.923 INFO:teuthology.orchestra.run.smithi007.stdout: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:42:41.923 INFO:teuthology.orchestra.run.smithi007.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-20T02:42:41.923 INFO:teuthology.orchestra.run.smithi007.stdout: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:42:41.923 INFO:teuthology.orchestra.run.smithi007.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-20T02:42:41.924 INFO:teuthology.orchestra.run.smithi007.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-20T02:42:41.924 INFO:teuthology.orchestra.run.smithi007.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-20T02:42:41.924 INFO:teuthology.orchestra.run.smithi007.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-20T02:42:41.924 INFO:teuthology.orchestra.run.smithi007.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-20T02:42:41.924 INFO:teuthology.orchestra.run.smithi007.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-20T02:42:41.924 INFO:teuthology.orchestra.run.smithi007.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-20T02:42:41.924 INFO:teuthology.orchestra.run.smithi007.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-20T02:42:41.925 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-20T02:42:41.925 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-20T02:42:41.925 INFO:teuthology.orchestra.run.smithi007.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-20T02:42:41.925 INFO:teuthology.orchestra.run.smithi007.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-20T02:42:41.925 INFO:teuthology.orchestra.run.smithi007.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-20T02:42:41.925 INFO:teuthology.orchestra.run.smithi007.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-20T02:42:41.925 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T02:42:41.925 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T02:42:41.926 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T02:42:41.926 INFO:teuthology.orchestra.run.smithi007.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-20T02:42:41.926 INFO:teuthology.orchestra.run.smithi007.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-20T02:42:41.926 INFO:teuthology.orchestra.run.smithi007.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-20T02:42:41.926 INFO:teuthology.orchestra.run.smithi007.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-20T02:42:41.926 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-20T02:42:41.926 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:42:41.927 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:42:41.927 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:42:41.927 INFO:teuthology.orchestra.run.smithi007.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-20T02:42:41.927 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-20T02:42:41.927 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-20T02:42:41.927 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-20T02:42:41.927 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-20T02:42:41.927 INFO:teuthology.orchestra.run.smithi007.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-20T02:42:41.928 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-20T02:42:41.928 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-20T02:42:41.928 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-20T02:42:41.928 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-20T02:42:41.928 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-20T02:42:41.928 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-20T02:42:41.928 INFO:teuthology.orchestra.run.smithi007.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-20T02:42:41.928 INFO:teuthology.orchestra.run.smithi007.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-20T02:42:41.929 INFO:teuthology.orchestra.run.smithi007.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-20T02:42:41.929 INFO:teuthology.orchestra.run.smithi007.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-20T02:42:41.929 INFO:teuthology.orchestra.run.smithi007.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-20T02:42:41.929 INFO:teuthology.orchestra.run.smithi007.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-20T02:42:41.929 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-20T02:42:41.929 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-20T02:42:41.929 INFO:teuthology.orchestra.run.smithi007.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-20T02:42:41.929 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-20T02:42:41.930 INFO:teuthology.orchestra.run.smithi007.stdout: python3-portend-2.6-1.el8.noarch 2023-07-20T02:42:41.930 INFO:teuthology.orchestra.run.smithi007.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-20T02:42:41.930 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-20T02:42:41.930 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-20T02:42:41.930 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-20T02:42:41.930 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-20T02:42:41.930 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-20T02:42:41.931 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:42:41.931 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:42:41.931 INFO:teuthology.orchestra.run.smithi007.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-20T02:42:41.931 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-20T02:42:41.931 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:42:41.931 INFO:teuthology.orchestra.run.smithi007.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-20T02:42:41.931 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-20T02:42:41.931 INFO:teuthology.orchestra.run.smithi007.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-20T02:42:41.932 INFO:teuthology.orchestra.run.smithi007.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-20T02:42:41.932 INFO:teuthology.orchestra.run.smithi007.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-20T02:42:41.932 INFO:teuthology.orchestra.run.smithi007.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-20T02:42:41.932 INFO:teuthology.orchestra.run.smithi007.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-20T02:42:41.932 INFO:teuthology.orchestra.run.smithi007.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-20T02:42:41.932 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-20T02:42:41.932 INFO:teuthology.orchestra.run.smithi007.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-20T02:42:41.932 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-20T02:42:41.933 INFO:teuthology.orchestra.run.smithi007.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-20T02:42:41.933 INFO:teuthology.orchestra.run.smithi007.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-20T02:42:41.933 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:42:41.933 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:42:41.933 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T02:42:41.933 INFO:teuthology.orchestra.run.smithi007.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-20T02:42:41.933 INFO:teuthology.orchestra.run.smithi007.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-20T02:42:41.933 INFO:teuthology.orchestra.run.smithi007.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-20T02:42:41.934 INFO:teuthology.orchestra.run.smithi007.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-20T02:42:41.934 INFO:teuthology.orchestra.run.smithi007.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-20T02:42:41.934 INFO:teuthology.orchestra.run.smithi007.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-20T02:42:41.934 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:42:41.934 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-07-20T02:42:42.227 DEBUG:teuthology.parallel:result is None 2023-07-20T02:42:42.227 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T02:42:42.649 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-20T02:42:42.708 INFO:teuthology.orchestra.run.smithi007.stdout:18.0.0-5040.g1b0b2eb0.el8 2023-07-20T02:42:42.708 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-5040.g1b0b2eb0.el8 2023-07-20T02:42:42.709 INFO:teuthology.task.install:The correct ceph version 18.0.0-5040.g1b0b2eb0 is installed. 2023-07-20T02:42:42.709 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T02:42:43.106 DEBUG:teuthology.orchestra.run.smithi158:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-20T02:42:43.136 INFO:teuthology.orchestra.run.smithi158.stdout:18.0.0-5040.g1b0b2eb0.el8 2023-07-20T02:42:43.136 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-5040.g1b0b2eb0.el8 2023-07-20T02:42:43.137 INFO:teuthology.task.install:The correct ceph version 18.0.0-5040.g1b0b2eb0 is installed. 2023-07-20T02:42:43.137 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T02:42:43.489 DEBUG:teuthology.orchestra.run.smithi188:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-20T02:42:43.517 INFO:teuthology.orchestra.run.smithi188.stdout:18.0.0-5040.g1b0b2eb0.el8 2023-07-20T02:42:43.518 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-5040.g1b0b2eb0.el8 2023-07-20T02:42:43.518 INFO:teuthology.task.install:The correct ceph version 18.0.0-5040.g1b0b2eb0 is installed. 2023-07-20T02:42:43.519 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-07-20T02:42:43.519 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-07-20T02:42:43.519 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-20T02:42:43.617 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-07-20T02:42:43.617 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-20T02:42:43.665 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2023-07-20T02:42:43.666 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-20T02:42:43.712 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-07-20T02:42:43.713 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-07-20T02:42:43.713 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/daemon-helper 2023-07-20T02:42:43.784 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-20T02:42:43.893 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-07-20T02:42:43.893 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/usr/bin/daemon-helper 2023-07-20T02:42:43.925 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-20T02:42:43.996 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2023-07-20T02:42:43.997 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/daemon-helper 2023-07-20T02:42:44.029 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-20T02:42:44.099 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-07-20T02:42:44.100 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-07-20T02:42:44.100 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-20T02:42:44.168 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-20T02:42:44.276 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-07-20T02:42:44.277 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-20T02:42:44.308 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-20T02:42:44.378 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2023-07-20T02:42:44.378 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-20T02:42:44.411 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-20T02:42:44.480 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-07-20T02:42:44.481 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-07-20T02:42:44.481 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/stdin-killer 2023-07-20T02:42:44.550 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-20T02:42:44.658 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-07-20T02:42:44.659 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/usr/bin/stdin-killer 2023-07-20T02:42:44.691 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-20T02:42:44.763 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2023-07-20T02:42:44.763 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/stdin-killer 2023-07-20T02:42:44.796 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-20T02:42:44.865 INFO:teuthology.run_tasks:Running task cephadm... 2023-07-20T02:42:44.961 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2023-07-20T02:42:44.961 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T02:42:44.973 INFO:tasks.cephadm:Cluster fsid is 1b04643c-26a7-11ee-9b34-001a4aab830c 2023-07-20T02:42:44.974 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-07-20T02:42:44.975 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.7', 'mon.b': '172.21.15.158', 'mon.c': '172.21.15.188'} 2023-07-20T02:42:44.975 INFO:tasks.cephadm:First mon is mon.a on smithi007 2023-07-20T02:42:44.976 INFO:tasks.cephadm:First mgr is x 2023-07-20T02:42:44.976 INFO:tasks.cephadm:Normalizing hostnames... 2023-07-20T02:42:44.977 DEBUG:teuthology.orchestra.run.smithi007:> sudo hostname $(hostname -s) 2023-07-20T02:42:45.050 DEBUG:teuthology.orchestra.run.smithi158:> sudo hostname $(hostname -s) 2023-07-20T02:42:45.082 DEBUG:teuthology.orchestra.run.smithi188:> sudo hostname $(hostname -s) 2023-07-20T02:42:45.115 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2023-07-20T02:42:45.115 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T02:42:45.506 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/71562/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.0.0-5040-g1b0b2eb0', 'node_name': '172.21.5.31+adami01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.0.0-5040.g1b0b2eb0'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2023-07-19 22:37:13.930734', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20230719.201220', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2023-07-20T02:42:45.865 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-pdonnell-testing-20230719.201220, sha1 1b0b2eb005a7c51d97a788b9900d511e1394b537 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T02:42:45.866 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/x86_64/flavors/default/cephadm 2023-07-20T02:42:45.866 DEBUG:teuthology.orchestra.run.smithi007:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-20T02:42:46.645 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 20 02:42 /home/ubuntu/cephtest/cephadm 2023-07-20T02:42:46.646 DEBUG:teuthology.orchestra.run.smithi158:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-20T02:42:47.581 INFO:teuthology.orchestra.run.smithi158.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 20 02:42 /home/ubuntu/cephtest/cephadm 2023-07-20T02:42:47.582 DEBUG:teuthology.orchestra.run.smithi188:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-20T02:42:48.672 INFO:teuthology.orchestra.run.smithi188.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 20 02:42 /home/ubuntu/cephtest/cephadm 2023-07-20T02:42:48.673 DEBUG:teuthology.orchestra.run.smithi007:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-20T02:42:48.724 DEBUG:teuthology.orchestra.run.smithi158:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-20T02:42:48.745 DEBUG:teuthology.orchestra.run.smithi188:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-20T02:42:48.782 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 on all hosts... 2023-07-20T02:42:48.782 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 pull 2023-07-20T02:42:48.788 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 pull 2023-07-20T02:42:48.790 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 pull 2023-07-20T02:42:50.082 INFO:teuthology.orchestra.run.smithi188.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T02:42:50.103 INFO:teuthology.orchestra.run.smithi158.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T02:42:51.471 INFO:teuthology.orchestra.run.smithi007.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T02:43:25.148 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-07-20T02:43:25.149 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_version": "ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev)", 2023-07-20T02:43:25.149 INFO:teuthology.orchestra.run.smithi007.stdout: "image_id": "3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7", 2023-07-20T02:43:25.149 INFO:teuthology.orchestra.run.smithi007.stdout: "repo_digests": [ 2023-07-20T02:43:25.149 INFO:teuthology.orchestra.run.smithi007.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5e9e7fcbd408a2723f648774346bcaf016d0aaddd533517186bfd5cb559518b1" 2023-07-20T02:43:25.149 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2023-07-20T02:43:25.149 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-07-20T02:43:37.785 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2023-07-20T02:43:37.786 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_version": "ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev)", 2023-07-20T02:43:37.786 INFO:teuthology.orchestra.run.smithi188.stdout: "image_id": "3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7", 2023-07-20T02:43:37.786 INFO:teuthology.orchestra.run.smithi188.stdout: "repo_digests": [ 2023-07-20T02:43:37.786 INFO:teuthology.orchestra.run.smithi188.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5e9e7fcbd408a2723f648774346bcaf016d0aaddd533517186bfd5cb559518b1" 2023-07-20T02:43:37.787 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2023-07-20T02:43:37.787 INFO:teuthology.orchestra.run.smithi188.stdout:} 2023-07-20T02:43:39.991 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-07-20T02:43:39.991 INFO:teuthology.orchestra.run.smithi158.stdout: "ceph_version": "ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev)", 2023-07-20T02:43:39.992 INFO:teuthology.orchestra.run.smithi158.stdout: "image_id": "3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7", 2023-07-20T02:43:39.992 INFO:teuthology.orchestra.run.smithi158.stdout: "repo_digests": [ 2023-07-20T02:43:39.992 INFO:teuthology.orchestra.run.smithi158.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5e9e7fcbd408a2723f648774346bcaf016d0aaddd533517186bfd5cb559518b1" 2023-07-20T02:43:39.992 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-07-20T02:43:39.992 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-07-20T02:43:40.011 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /etc/ceph 2023-07-20T02:43:40.091 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /etc/ceph 2023-07-20T02:43:40.124 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /etc/ceph 2023-07-20T02:43:40.163 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 777 /etc/ceph 2023-07-20T02:43:40.236 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 777 /etc/ceph 2023-07-20T02:43:40.264 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 777 /etc/ceph 2023-07-20T02:43:40.297 INFO:tasks.cephadm:Writing seed config... 2023-07-20T02:43:40.298 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-07-20T02:43:40.298 INFO:tasks.cephadm: override: [client] debug client = 20 2023-07-20T02:43:40.298 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-07-20T02:43:40.299 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-07-20T02:43:40.299 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-07-20T02:43:40.299 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2023-07-20T02:43:40.299 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-07-20T02:43:40.299 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-07-20T02:43:40.300 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-07-20T02:43:40.300 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2023-07-20T02:43:40.300 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2023-07-20T02:43:40.300 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2023-07-20T02:43:40.300 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2023-07-20T02:43:40.301 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-07-20T02:43:40.301 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-07-20T02:43:40.301 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-07-20T02:43:40.301 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-07-20T02:43:40.301 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-07-20T02:43:40.302 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-07-20T02:43:40.302 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-07-20T02:43:40.302 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2023-07-20T02:43:40.302 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-07-20T02:43:40.302 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-07-20T02:43:40.303 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-07-20T02:43:40.303 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-07-20T02:43:40.303 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-07-20T02:43:40.303 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-07-20T02:43:40.303 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-07-20T02:43:40.304 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-07-20T02:43:40.304 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2023-07-20T02:43:40.304 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2023-07-20T02:43:40.304 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-07-20T02:43:40.304 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-07-20T02:43:40.305 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2023-07-20T02:43:40.305 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-07-20T02:43:40.305 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-07-20T02:43:40.305 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-07-20T02:43:40.306 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-07-20T02:43:40.307 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-07-20T02:43:40.307 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-07-20T02:43:40.308 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2023-07-20T02:43:40.308 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2023-07-20T02:43:40.310 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-07-20T02:43:40.310 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-07-20T02:43:40.355 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 1b04643c-26a7-11ee-9b34-001a4aab830c ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-07-20T02:43:40.355 DEBUG:teuthology.orchestra.run.smithi007:mon.a> sudo journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mon.a.service 2023-07-20T02:43:40.399 DEBUG:teuthology.orchestra.run.smithi007:mgr.x> sudo journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mgr.x.service 2023-07-20T02:43:40.445 INFO:tasks.cephadm:Bootstrapping... 2023-07-20T02:43:40.445 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 -v bootstrap --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.7 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-07-20T02:43:40.468 INFO:journalctl@ceph.mon.a.smithi007.stdout:-- Logs begin at Thu 2023-07-20 02:22:32 UTC. -- 2023-07-20T02:43:40.514 INFO:journalctl@ceph.mgr.x.smithi007.stdout:-- Logs begin at Thu 2023-07-20 02:22:32 UTC. -- 2023-07-20T02:43:40.675 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-07-20T02:43:40.676 INFO:teuthology.orchestra.run.smithi007.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537', '-v', 'bootstrap', '--fsid', '1b04643c-26a7-11ee-9b34-001a4aab830c', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.7', '--skip-admin-label'] 2023-07-20T02:43:40.851 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stdout 4.1.1 2023-07-20T02:43:40.851 INFO:teuthology.orchestra.run.smithi007.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-07-20T02:43:40.852 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying podman|docker is present... 2023-07-20T02:43:41.023 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stdout 4.1.1 2023-07-20T02:43:41.024 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying lvm2 is present... 2023-07-20T02:43:41.024 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying time synchronization is in place... 2023-07-20T02:43:41.038 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-07-20T02:43:41.038 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-07-20T02:43:41.051 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-07-20T02:43:41.051 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2023-07-20T02:43:41.064 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout enabled 2023-07-20T02:43:41.080 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout active 2023-07-20T02:43:41.080 INFO:teuthology.orchestra.run.smithi007.stdout:Unit chronyd.service is enabled and running 2023-07-20T02:43:41.080 INFO:teuthology.orchestra.run.smithi007.stdout:Repeating the final host check... 2023-07-20T02:43:41.248 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stdout 4.1.1 2023-07-20T02:43:41.249 INFO:teuthology.orchestra.run.smithi007.stdout:podman (/bin/podman) version 4.1.1 is present 2023-07-20T02:43:41.249 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl is present 2023-07-20T02:43:41.249 INFO:teuthology.orchestra.run.smithi007.stdout:lvcreate is present 2023-07-20T02:43:41.262 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-07-20T02:43:41.262 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-07-20T02:43:41.275 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-07-20T02:43:41.275 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2023-07-20T02:43:41.289 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout enabled 2023-07-20T02:43:41.302 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout active 2023-07-20T02:43:41.302 INFO:teuthology.orchestra.run.smithi007.stdout:Unit chronyd.service is enabled and running 2023-07-20T02:43:41.302 INFO:teuthology.orchestra.run.smithi007.stdout:Host looks OK 2023-07-20T02:43:41.303 INFO:teuthology.orchestra.run.smithi007.stdout:Cluster fsid: 1b04643c-26a7-11ee-9b34-001a4aab830c 2023-07-20T02:43:41.303 INFO:teuthology.orchestra.run.smithi007.stdout:Acquiring lock 140143378967016 on /run/cephadm/1b04643c-26a7-11ee-9b34-001a4aab830c.lock 2023-07-20T02:43:41.303 INFO:teuthology.orchestra.run.smithi007.stdout:Lock 140143378967016 acquired on /run/cephadm/1b04643c-26a7-11ee-9b34-001a4aab830c.lock 2023-07-20T02:43:41.304 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying IP 172.21.15.7 port 3300 ... 2023-07-20T02:43:41.304 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying IP 172.21.15.7 port 6789 ... 2023-07-20T02:43:41.304 INFO:teuthology.orchestra.run.smithi007.stdout:Base mon IP(s) is [172.21.15.7:3300, 172.21.15.7:6789], mon addrv is [v2:172.21.15.7:3300,v1:172.21.15.7:6789] 2023-07-20T02:43:41.311 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp metric 100 2023-07-20T02:43:41.311 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.7 metric 100 2023-07-20T02:43:41.318 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-07-20T02:43:41.318 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2023-07-20T02:43:41.318 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1785sec hoplimit 64 pref medium 2023-07-20T02:43:41.324 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-07-20T02:43:41.325 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-07-20T02:43:41.325 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-07-20T02:43:41.325 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2023-07-20T02:43:41.325 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15ea/64 scope link 2023-07-20T02:43:41.325 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-07-20T02:43:41.326 INFO:teuthology.orchestra.run.smithi007.stdout:Mon IP `172.21.15.7` is in CIDR network `172.21.0.0/20` 2023-07-20T02:43:41.326 INFO:teuthology.orchestra.run.smithi007.stdout:Mon IP `172.21.15.7` is in CIDR network `172.21.0.0/20` 2023-07-20T02:43:41.326 INFO:teuthology.orchestra.run.smithi007.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-07-20T02:43:41.327 INFO:teuthology.orchestra.run.smithi007.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-07-20T02:43:41.327 INFO:teuthology.orchestra.run.smithi007.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T02:43:41.800 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stdout 3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7 2023-07-20T02:43:41.800 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T02:43:41.801 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Getting image source signatures 2023-07-20T02:43:41.801 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Copying blob sha256:9a3af2d1f966fade87cf08b715899d7eff2dde16dabc6e2e4c3ec2059da8adc1 2023-07-20T02:43:41.801 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Copying blob sha256:6022c611d474f2b306ddd1edfc8360123ece5266119096f25fde843d7967e636 2023-07-20T02:43:41.802 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Copying config sha256:3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7 2023-07-20T02:43:41.802 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Writing manifest to image destination 2023-07-20T02:43:41.802 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Storing signatures 2023-07-20T02:43:42.675 INFO:teuthology.orchestra.run.smithi007.stdout:ceph: stdout ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev) 2023-07-20T02:43:42.676 INFO:teuthology.orchestra.run.smithi007.stdout:Ceph version: ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev) 2023-07-20T02:43:42.676 INFO:teuthology.orchestra.run.smithi007.stdout:Extracting ceph user uid/gid from container image... 2023-07-20T02:43:43.453 INFO:teuthology.orchestra.run.smithi007.stdout:stat: stdout 167 167 2023-07-20T02:43:43.453 INFO:teuthology.orchestra.run.smithi007.stdout:Creating initial keys... 2023-07-20T02:43:44.228 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph-authtool: stdout AQBfn7hk+tRbMxAARolbS3WhoYtQsv6L8auBng== 2023-07-20T02:43:45.105 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph-authtool: stdout AQBgn7hko9RwJhAABR+cN/hsRSg5UDo3ocTodw== 2023-07-20T02:43:45.982 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph-authtool: stdout AQBhn7hk8yiSHRAAkp8OjkpeNiUSBI7mlbdRvA== 2023-07-20T02:43:45.983 INFO:teuthology.orchestra.run.smithi007.stdout:Creating initial monmap... 2023-07-20T02:43:46.863 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-20T02:43:46.863 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2023-07-20T02:43:46.863 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 1b04643c-26a7-11ee-9b34-001a4aab830c 2023-07-20T02:43:46.863 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-20T02:43:46.864 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool for a [v2:172.21.15.7:3300,v1:172.21.15.7:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-20T02:43:46.864 INFO:teuthology.orchestra.run.smithi007.stdout:setting min_mon_release = pacific 2023-07-20T02:43:46.864 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: set fsid to 1b04643c-26a7-11ee-9b34-001a4aab830c 2023-07-20T02:43:46.864 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-20T02:43:46.864 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:43:46.865 INFO:teuthology.orchestra.run.smithi007.stdout:Creating mon... 2023-07-20T02:43:47.740 INFO:teuthology.orchestra.run.smithi007.stdout:create mon.a on 2023-07-20T02:43:47.992 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-07-20T02:43:48.193 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-07-20T02:43:50.379 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-1b04643c-26a7-11ee-9b34-001a4aab830c.target → /etc/systemd/system/ceph-1b04643c-26a7-11ee-9b34-001a4aab830c.target. 2023-07-20T02:43:50.379 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-1b04643c-26a7-11ee-9b34-001a4aab830c.target → /etc/systemd/system/ceph-1b04643c-26a7-11ee-9b34-001a4aab830c.target. 2023-07-20T02:43:53.250 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mon.a 2023-07-20T02:43:53.251 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Failed to reset failed state of unit ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mon.a.service: Unit ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mon.a.service not loaded. 2023-07-20T02:43:53.466 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-1b04643c-26a7-11ee-9b34-001a4aab830c.target.wants/ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mon.a.service → /etc/systemd/system/ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@.service. 2023-07-20T02:43:53.692 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:43:53 smithi007 systemd[1]: Starting Ceph mon.a for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:43:56.442 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:43:56 smithi007 podman[125805]: 2023-07-20T02:43:56.597 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-20T02:43:56.597 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout disabled 2023-07-20T02:43:56.613 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-20T02:43:56.613 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2023-07-20T02:43:56.614 INFO:teuthology.orchestra.run.smithi007.stdout:firewalld.service is not enabled 2023-07-20T02:43:56.614 INFO:teuthology.orchestra.run.smithi007.stdout:Not possible to enable service . firewalld.service is not available 2023-07-20T02:43:56.614 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mon to start... 2023-07-20T02:43:56.614 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mon... 2023-07-20T02:43:56.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:43:56 smithi007 bash[125805]: 64929a833ebceb049425fac552759551c8f846b9c66e04e728dccb1956ef72e0 2023-07-20T02:43:56.890 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:43:56 smithi007 ceph-mon[125845]: mkfs 1b04643c-26a7-11ee-9b34-001a4aab830c 2023-07-20T02:43:56.890 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:43:56 smithi007 ceph-mon[125845]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-20T02:43:56.890 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:43:56 smithi007 systemd[1]: Started Ceph mon.a for 1b04643c-26a7-11ee-9b34-001a4aab830c. 2023-07-20T02:43:57.701 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout cluster: 2023-07-20T02:43:57.702 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout id: 1b04643c-26a7-11ee-9b34-001a4aab830c 2023-07-20T02:43:57.702 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-07-20T02:43:57.702 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2023-07-20T02:43:57.702 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout services: 2023-07-20T02:43:57.702 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.687396s) 2023-07-20T02:43:57.702 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-07-20T02:43:57.702 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-07-20T02:43:57.703 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2023-07-20T02:43:57.703 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout data: 2023-07-20T02:43:57.703 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-07-20T02:43:57.703 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-07-20T02:43:57.703 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-07-20T02:43:57.703 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout pgs: 2023-07-20T02:43:57.703 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2023-07-20T02:43:57.704 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.144+0000 7f3dcebec700 1 Processor -- start 2023-07-20T02:43:57.704 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.144+0000 7f3dcebec700 1 -- start start 2023-07-20T02:43:57.704 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.144+0000 7f3dcebec700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3dc807be50 0x7f3dc807a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:43:57.704 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.144+0000 7f3dcebec700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc807a880 con 0x7f3dc807be50 2023-07-20T02:43:57.704 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.146+0000 7f3dcc988700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3dc807be50 0x7f3dc807a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:43:57.704 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.146+0000 7f3dcc988700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3dc807be50 0x7f3dc807a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51872/0 (socket says 172.21.15.7:51872) 2023-07-20T02:43:57.705 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.146+0000 7f3dcc988700 1 -- 172.21.15.7:0/2783112173 learned_addr learned my addr 172.21.15.7:0/2783112173 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:43:57.705 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.146+0000 7f3dcc988700 1 -- 172.21.15.7:0/2783112173 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc807a9c0 con 0x7f3dc807be50 2023-07-20T02:43:57.705 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.146+0000 7f3dcc988700 1 --2- 172.21.15.7:0/2783112173 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3dc807be50 0x7f3dc807a2b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc0098a0 tx=0x7f3dbc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=425904553af40a39 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:43:57.705 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.146+0000 7f3dc77fe700 1 -- 172.21.15.7:0/2783112173 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3dbc004160 con 0x7f3dc807be50 2023-07-20T02:43:57.705 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.146+0000 7f3dc77fe700 1 -- 172.21.15.7:0/2783112173 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dbc0042c0 con 0x7f3dc807be50 2023-07-20T02:43:57.705 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.147+0000 7f3dc77fe700 1 -- 172.21.15.7:0/2783112173 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3dbc0045a0 con 0x7f3dc807be50 2023-07-20T02:43:57.705 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.147+0000 7f3dcebec700 1 -- 172.21.15.7:0/2783112173 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3dc807be50 msgr2=0x7f3dc807a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:43:57.706 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.147+0000 7f3dcebec700 1 --2- 172.21.15.7:0/2783112173 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3dc807be50 0x7f3dc807a2b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc0098a0 tx=0x7f3dbc009bb0 comp rx=0 tx=0).stop 2023-07-20T02:43:57.706 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.147+0000 7f3dcebec700 1 -- 172.21.15.7:0/2783112173 shutdown_connections 2023-07-20T02:43:57.706 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.147+0000 7f3dcebec700 1 --2- 172.21.15.7:0/2783112173 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3dc807be50 0x7f3dc807a2b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:43:57.706 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.147+0000 7f3dcebec700 1 -- 172.21.15.7:0/2783112173 >> 172.21.15.7:0/2783112173 conn(0x7f3dc8101440 msgr2=0x7f3dc8103860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:43:57.706 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.147+0000 7f3dcebec700 1 -- 172.21.15.7:0/2783112173 shutdown_connections 2023-07-20T02:43:57.706 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.147+0000 7f3dcebec700 1 -- 172.21.15.7:0/2783112173 wait complete. 2023-07-20T02:43:57.706 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.148+0000 7f3dcebec700 1 Processor -- start 2023-07-20T02:43:57.706 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.148+0000 7f3dcebec700 1 -- start start 2023-07-20T02:43:57.707 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.149+0000 7f3dcebec700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3dc807be50 0x7f3dc819b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:43:57.707 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.149+0000 7f3dcebec700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc819baa0 con 0x7f3dc807be50 2023-07-20T02:43:57.707 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.149+0000 7f3dcc988700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3dc807be50 0x7f3dc819b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:43:57.707 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.149+0000 7f3dcc988700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3dc807be50 0x7f3dc819b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51874/0 (socket says 172.21.15.7:51874) 2023-07-20T02:43:57.707 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.149+0000 7f3dcc988700 1 -- 172.21.15.7:0/251943187 learned_addr learned my addr 172.21.15.7:0/251943187 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:43:57.707 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.149+0000 7f3dcc988700 1 -- 172.21.15.7:0/251943187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dbc009550 con 0x7f3dc807be50 2023-07-20T02:43:57.707 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.150+0000 7f3dcc988700 1 --2- 172.21.15.7:0/251943187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3dc807be50 0x7f3dc819b560 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f3dc807b320 tx=0x7f3dbc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:43:57.708 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.150+0000 7f3dc5ffb700 1 -- 172.21.15.7:0/251943187 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3dbc011630 con 0x7f3dc807be50 2023-07-20T02:43:57.708 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.150+0000 7f3dc5ffb700 1 -- 172.21.15.7:0/251943187 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dbc011790 con 0x7f3dc807be50 2023-07-20T02:43:57.708 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.150+0000 7f3dc5ffb700 1 -- 172.21.15.7:0/251943187 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3dbc011a30 con 0x7f3dc807be50 2023-07-20T02:43:57.708 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.150+0000 7f3dcebec700 1 -- 172.21.15.7:0/251943187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dc819bca0 con 0x7f3dc807be50 2023-07-20T02:43:57.708 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.150+0000 7f3dcebec700 1 -- 172.21.15.7:0/251943187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dc819c0c0 con 0x7f3dc807be50 2023-07-20T02:43:57.708 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.150+0000 7f3dc5ffb700 1 -- 172.21.15.7:0/251943187 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f3dbc022070 con 0x7f3dc807be50 2023-07-20T02:43:57.709 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.151+0000 7f3dc5ffb700 1 -- 172.21.15.7:0/251943187 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3dbc01c450 con 0x7f3dc807be50 2023-07-20T02:43:57.709 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.151+0000 7f3dcebec700 1 -- 172.21.15.7:0/251943187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dc8195440 con 0x7f3dc807be50 2023-07-20T02:43:57.709 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.154+0000 7f3dc5ffb700 1 -- 172.21.15.7:0/251943187 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f3dbc01b450 con 0x7f3dc807be50 2023-07-20T02:43:57.709 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.230+0000 7f3dcebec700 1 -- 172.21.15.7:0/251943187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f3dc802c970 con 0x7f3dc807be50 2023-07-20T02:43:57.709 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.230+0000 7f3dc5ffb700 1 -- 172.21.15.7:0/251943187 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f3dbc01b5f0 con 0x7f3dc807be50 2023-07-20T02:43:57.709 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.231+0000 7f3dcebec700 1 -- 172.21.15.7:0/251943187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3dc807be50 msgr2=0x7f3dc819b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:43:57.709 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.231+0000 7f3dcebec700 1 --2- 172.21.15.7:0/251943187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3dc807be50 0x7f3dc819b560 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f3dc807b320 tx=0x7f3dbc004750 comp rx=0 tx=0).stop 2023-07-20T02:43:57.710 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.231+0000 7f3dcebec700 1 -- 172.21.15.7:0/251943187 shutdown_connections 2023-07-20T02:43:57.710 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.231+0000 7f3dcebec700 1 --2- 172.21.15.7:0/251943187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3dc807be50 0x7f3dc819b560 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:43:57.710 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.231+0000 7f3dcebec700 1 -- 172.21.15.7:0/251943187 >> 172.21.15.7:0/251943187 conn(0x7f3dc8101440 msgr2=0x7f3dc8102dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:43:57.710 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.232+0000 7f3dcebec700 1 -- 172.21.15.7:0/251943187 shutdown_connections 2023-07-20T02:43:57.710 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:57.232+0000 7f3dcebec700 1 -- 172.21.15.7:0/251943187 wait complete. 2023-07-20T02:43:57.710 INFO:teuthology.orchestra.run.smithi007.stdout:mon is available 2023-07-20T02:43:57.710 INFO:teuthology.orchestra.run.smithi007.stdout:Assimilating anything we can from ceph.conf... 2023-07-20T02:43:57.942 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:43:57 smithi007 ceph-mon[125845]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-20T02:43:57.942 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:43:57 smithi007 ceph-mon[125845]: monmap e1: 1 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0]} removed_ranks: {} 2023-07-20T02:43:57.942 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:43:57 smithi007 ceph-mon[125845]: fsmap 2023-07-20T02:43:57.942 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:43:57 smithi007 ceph-mon[125845]: osdmap e1: 0 total, 0 up, 0 in 2023-07-20T02:43:57.942 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:43:57 smithi007 ceph-mon[125845]: mgrmap e1: no daemons active 2023-07-20T02:43:57.943 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:43:57 smithi007 ceph-mon[125845]: from='client.? 172.21.15.7:0/251943187' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-07-20T02:43:58.672 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:43:58 smithi007 ceph-mon[125845]: from='client.? 172.21.15.7:0/752614983' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-20T02:43:58.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:43:58 smithi007 ceph-mon[125845]: from='client.? 172.21.15.7:0/752614983' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-07-20T02:43:58.684 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2023-07-20T02:43:58.685 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [global] 2023-07-20T02:43:58.685 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout fsid = 1b04643c-26a7-11ee-9b34-001a4aab830c 2023-07-20T02:43:58.685 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.7:3300,v1:172.21.15.7:6789] 2023-07-20T02:43:58.686 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-07-20T02:43:58.686 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-07-20T02:43:58.686 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-07-20T02:43:58.686 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-07-20T02:43:58.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2023-07-20T02:43:58.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [mgr] 2023-07-20T02:43:58.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-07-20T02:43:58.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2023-07-20T02:43:58.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [osd] 2023-07-20T02:43:58.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-07-20T02:43:58.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-07-20T02:43:58.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-07-20T02:43:58.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-07-20T02:43:58.689 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-07-20T02:43:58.689 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.209+0000 7fccd8f9e700 1 Processor -- start 2023-07-20T02:43:58.689 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.209+0000 7fccd8f9e700 1 -- start start 2023-07-20T02:43:58.689 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.210+0000 7fccd8f9e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccd407be50 0x7fccd407a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:43:58.690 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.210+0000 7fccd8f9e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd407a880 con 0x7fccd407be50 2023-07-20T02:43:58.690 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.210+0000 7fccd259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccd407be50 0x7fccd407a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:43:58.690 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.210+0000 7fccd259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccd407be50 0x7fccd407a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51878/0 (socket says 172.21.15.7:51878) 2023-07-20T02:43:58.690 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.210+0000 7fccd259c700 1 -- 172.21.15.7:0/28973388 learned_addr learned my addr 172.21.15.7:0/28973388 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:43:58.690 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.211+0000 7fccd259c700 1 -- 172.21.15.7:0/28973388 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd407a9c0 con 0x7fccd407be50 2023-07-20T02:43:58.691 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.211+0000 7fccd259c700 1 --2- 172.21.15.7:0/28973388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccd407be50 0x7fccd407a2b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fccc40098a0 tx=0x7fccc4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=721dca5c3b2b918a server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:43:58.691 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.211+0000 7fccd159a700 1 -- 172.21.15.7:0/28973388 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fccc4004160 con 0x7fccd407be50 2023-07-20T02:43:58.691 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.211+0000 7fccd159a700 1 -- 172.21.15.7:0/28973388 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc40042c0 con 0x7fccd407be50 2023-07-20T02:43:58.691 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.211+0000 7fccd159a700 1 -- 172.21.15.7:0/28973388 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fccc40045a0 con 0x7fccd407be50 2023-07-20T02:43:58.691 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.212+0000 7fccd8f9e700 1 -- 172.21.15.7:0/28973388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccd407be50 msgr2=0x7fccd407a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:43:58.692 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.212+0000 7fccd8f9e700 1 --2- 172.21.15.7:0/28973388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccd407be50 0x7fccd407a2b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fccc40098a0 tx=0x7fccc4009bb0 comp rx=0 tx=0).stop 2023-07-20T02:43:58.692 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.212+0000 7fccd8f9e700 1 -- 172.21.15.7:0/28973388 shutdown_connections 2023-07-20T02:43:58.692 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.212+0000 7fccd8f9e700 1 --2- 172.21.15.7:0/28973388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccd407be50 0x7fccd407a2b0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:43:58.692 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.212+0000 7fccd8f9e700 1 -- 172.21.15.7:0/28973388 >> 172.21.15.7:0/28973388 conn(0x7fccd4101440 msgr2=0x7fccd4103860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:43:58.693 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.212+0000 7fccd8f9e700 1 -- 172.21.15.7:0/28973388 shutdown_connections 2023-07-20T02:43:58.693 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.212+0000 7fccd8f9e700 1 -- 172.21.15.7:0/28973388 wait complete. 2023-07-20T02:43:58.693 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.213+0000 7fccd8f9e700 1 Processor -- start 2023-07-20T02:43:58.693 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.213+0000 7fccd8f9e700 1 -- start start 2023-07-20T02:43:58.693 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.213+0000 7fccd8f9e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccd407be50 0x7fccd419b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:43:58.694 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.213+0000 7fccd8f9e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd419baa0 con 0x7fccd407be50 2023-07-20T02:43:58.694 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.214+0000 7fccd259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccd407be50 0x7fccd419b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:43:58.694 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.214+0000 7fccd259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccd407be50 0x7fccd419b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51892/0 (socket says 172.21.15.7:51892) 2023-07-20T02:43:58.694 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.214+0000 7fccd259c700 1 -- 172.21.15.7:0/752614983 learned_addr learned my addr 172.21.15.7:0/752614983 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:43:58.694 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.214+0000 7fccd259c700 1 -- 172.21.15.7:0/752614983 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc4009550 con 0x7fccd407be50 2023-07-20T02:43:58.695 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.214+0000 7fccd259c700 1 --2- 172.21.15.7:0/752614983 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccd407be50 0x7fccd419b560 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fccd407b320 tx=0x7fccc4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:43:58.695 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.214+0000 7fccc37fe700 1 -- 172.21.15.7:0/752614983 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fccc4011630 con 0x7fccd407be50 2023-07-20T02:43:58.695 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.214+0000 7fccc37fe700 1 -- 172.21.15.7:0/752614983 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc4011790 con 0x7fccd407be50 2023-07-20T02:43:58.695 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.214+0000 7fccc37fe700 1 -- 172.21.15.7:0/752614983 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fccc4011a30 con 0x7fccd407be50 2023-07-20T02:43:58.696 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.214+0000 7fccd8f9e700 1 -- 172.21.15.7:0/752614983 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccd419bca0 con 0x7fccd407be50 2023-07-20T02:43:58.696 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.215+0000 7fccd8f9e700 1 -- 172.21.15.7:0/752614983 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccd419c0c0 con 0x7fccd407be50 2023-07-20T02:43:58.696 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.215+0000 7fccc37fe700 1 -- 172.21.15.7:0/752614983 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fccc4022070 con 0x7fccd407be50 2023-07-20T02:43:58.696 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.215+0000 7fccc37fe700 1 -- 172.21.15.7:0/752614983 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fccc401c450 con 0x7fccd407be50 2023-07-20T02:43:58.696 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.216+0000 7fccd8f9e700 1 -- 172.21.15.7:0/752614983 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccd4195440 con 0x7fccd407be50 2023-07-20T02:43:58.697 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.218+0000 7fccc37fe700 1 -- 172.21.15.7:0/752614983 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fccc401b450 con 0x7fccd407be50 2023-07-20T02:43:58.697 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.290+0000 7fccd8f9e700 1 -- 172.21.15.7:0/752614983 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fccd402cbf0 con 0x7fccd407be50 2023-07-20T02:43:58.697 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.295+0000 7fccc37fe700 1 -- 172.21.15.7:0/752614983 <== mon.0 v2:172.21.15.7:3300/0 7 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fccc401b5f0 con 0x7fccd407be50 2023-07-20T02:43:58.697 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.295+0000 7fccc37fe700 1 -- 172.21.15.7:0/752614983 <== mon.0 v2:172.21.15.7:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+427 (secure 0 0 0) 0x7fccc40375f0 con 0x7fccd407be50 2023-07-20T02:43:58.697 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.296+0000 7fccd8f9e700 1 -- 172.21.15.7:0/752614983 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccd407be50 msgr2=0x7fccd419b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:43:58.698 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.296+0000 7fccd8f9e700 1 --2- 172.21.15.7:0/752614983 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccd407be50 0x7fccd419b560 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fccd407b320 tx=0x7fccc4004750 comp rx=0 tx=0).stop 2023-07-20T02:43:58.698 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.296+0000 7fccd8f9e700 1 -- 172.21.15.7:0/752614983 shutdown_connections 2023-07-20T02:43:58.698 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.296+0000 7fccd8f9e700 1 --2- 172.21.15.7:0/752614983 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccd407be50 0x7fccd419b560 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:43:58.698 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.296+0000 7fccd8f9e700 1 -- 172.21.15.7:0/752614983 >> 172.21.15.7:0/752614983 conn(0x7fccd4101440 msgr2=0x7fccd4102dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:43:58.698 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.296+0000 7fccd8f9e700 1 -- 172.21.15.7:0/752614983 shutdown_connections 2023-07-20T02:43:58.699 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:58.296+0000 7fccd8f9e700 1 -- 172.21.15.7:0/752614983 wait complete. 2023-07-20T02:43:58.699 INFO:teuthology.orchestra.run.smithi007.stdout:Generating new minimal ceph.conf... 2023-07-20T02:43:59.692 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:43:59 smithi007 ceph-mon[125845]: from='client.? 172.21.15.7:0/399532624' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:43:59.761 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.209+0000 7f6ee4cc2700 1 Processor -- start 2023-07-20T02:43:59.761 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.209+0000 7f6ee4cc2700 1 -- start start 2023-07-20T02:43:59.761 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.209+0000 7f6ee4cc2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ee007be30 0x7f6ee007a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:43:59.761 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.209+0000 7f6ee4cc2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee007a860 con 0x7f6ee007be30 2023-07-20T02:43:59.761 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.209+0000 7f6ede59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ee007be30 0x7f6ee007a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:43:59.762 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.209+0000 7f6ede59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ee007be30 0x7f6ee007a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51896/0 (socket says 172.21.15.7:51896) 2023-07-20T02:43:59.762 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.209+0000 7f6ede59c700 1 -- 172.21.15.7:0/258508023 learned_addr learned my addr 172.21.15.7:0/258508023 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:43:59.762 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.210+0000 7f6ede59c700 1 -- 172.21.15.7:0/258508023 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee007a9a0 con 0x7f6ee007be30 2023-07-20T02:43:59.762 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.210+0000 7f6ede59c700 1 --2- 172.21.15.7:0/258508023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ee007be30 0x7f6ee007a290 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6ed00098a0 tx=0x7f6ed0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a9ea9fa82da0785 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:43:59.762 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.210+0000 7f6edd59a700 1 -- 172.21.15.7:0/258508023 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ed0004160 con 0x7f6ee007be30 2023-07-20T02:43:59.762 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.210+0000 7f6edd59a700 1 -- 172.21.15.7:0/258508023 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ed00042c0 con 0x7f6ee007be30 2023-07-20T02:43:59.762 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.210+0000 7f6edd59a700 1 -- 172.21.15.7:0/258508023 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ed0011a00 con 0x7f6ee007be30 2023-07-20T02:43:59.763 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.211+0000 7f6ee4cc2700 1 -- 172.21.15.7:0/258508023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ee007be30 msgr2=0x7f6ee007a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:43:59.763 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.211+0000 7f6ee4cc2700 1 --2- 172.21.15.7:0/258508023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ee007be30 0x7f6ee007a290 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6ed00098a0 tx=0x7f6ed0009bb0 comp rx=0 tx=0).stop 2023-07-20T02:43:59.763 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.211+0000 7f6ee4cc2700 1 -- 172.21.15.7:0/258508023 shutdown_connections 2023-07-20T02:43:59.763 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.211+0000 7f6ee4cc2700 1 --2- 172.21.15.7:0/258508023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ee007be30 0x7f6ee007a290 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:43:59.763 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.211+0000 7f6ee4cc2700 1 -- 172.21.15.7:0/258508023 >> 172.21.15.7:0/258508023 conn(0x7f6ee0101440 msgr2=0x7f6ee0103860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:43:59.763 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.211+0000 7f6ee4cc2700 1 -- 172.21.15.7:0/258508023 shutdown_connections 2023-07-20T02:43:59.763 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.211+0000 7f6ee4cc2700 1 -- 172.21.15.7:0/258508023 wait complete. 2023-07-20T02:43:59.763 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.212+0000 7f6ee4cc2700 1 Processor -- start 2023-07-20T02:43:59.764 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.212+0000 7f6ee4cc2700 1 -- start start 2023-07-20T02:43:59.764 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.213+0000 7f6ee4cc2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ee007be30 0x7f6ee019bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:43:59.764 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.213+0000 7f6ee4cc2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee019c270 con 0x7f6ee007be30 2023-07-20T02:43:59.764 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.213+0000 7f6ede59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ee007be30 0x7f6ee019bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:43:59.764 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.213+0000 7f6ede59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ee007be30 0x7f6ee019bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51910/0 (socket says 172.21.15.7:51910) 2023-07-20T02:43:59.764 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.213+0000 7f6ede59c700 1 -- 172.21.15.7:0/399532624 learned_addr learned my addr 172.21.15.7:0/399532624 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:43:59.764 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.213+0000 7f6ede59c700 1 -- 172.21.15.7:0/399532624 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed0009550 con 0x7f6ee007be30 2023-07-20T02:43:59.765 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.213+0000 7f6ede59c700 1 --2- 172.21.15.7:0/399532624 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ee007be30 0x7f6ee019bd30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6ed0009fb0 tx=0x7f6ed0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:43:59.765 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.214+0000 7f6ecf7fe700 1 -- 172.21.15.7:0/399532624 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ed0003930 con 0x7f6ee007be30 2023-07-20T02:43:59.765 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.214+0000 7f6ecf7fe700 1 -- 172.21.15.7:0/399532624 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ed0003a90 con 0x7f6ee007be30 2023-07-20T02:43:59.765 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.214+0000 7f6ecf7fe700 1 -- 172.21.15.7:0/399532624 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ed0019610 con 0x7f6ee007be30 2023-07-20T02:43:59.765 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.214+0000 7f6ee4cc2700 1 -- 172.21.15.7:0/399532624 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ee019c470 con 0x7f6ee007be30 2023-07-20T02:43:59.765 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.214+0000 7f6ee4cc2700 1 -- 172.21.15.7:0/399532624 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ee019c890 con 0x7f6ee007be30 2023-07-20T02:43:59.765 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.214+0000 7f6ecf7fe700 1 -- 172.21.15.7:0/399532624 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f6ed0019b80 con 0x7f6ee007be30 2023-07-20T02:43:59.765 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.214+0000 7f6ecf7fe700 1 -- 172.21.15.7:0/399532624 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6ed001a430 con 0x7f6ee007be30 2023-07-20T02:43:59.766 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.215+0000 7f6ee4cc2700 1 -- 172.21.15.7:0/399532624 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ee0061da0 con 0x7f6ee007be30 2023-07-20T02:43:59.766 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.218+0000 7f6ecf7fe700 1 -- 172.21.15.7:0/399532624 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f6ed002de40 con 0x7f6ee007be30 2023-07-20T02:43:59.766 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.291+0000 7f6ee4cc2700 1 -- 172.21.15.7:0/399532624 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f6ee019f3c0 con 0x7f6ee007be30 2023-07-20T02:43:59.766 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.291+0000 7f6ecf7fe700 1 -- 172.21.15.7:0/399532624 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+173 (secure 0 0 0) 0x7f6ed0003c00 con 0x7f6ee007be30 2023-07-20T02:43:59.766 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.292+0000 7f6ee4cc2700 1 -- 172.21.15.7:0/399532624 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ee007be30 msgr2=0x7f6ee019bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:43:59.766 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.292+0000 7f6ee4cc2700 1 --2- 172.21.15.7:0/399532624 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ee007be30 0x7f6ee019bd30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6ed0009fb0 tx=0x7f6ed0011ad0 comp rx=0 tx=0).stop 2023-07-20T02:43:59.766 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.292+0000 7f6ee4cc2700 1 -- 172.21.15.7:0/399532624 shutdown_connections 2023-07-20T02:43:59.767 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.292+0000 7f6ee4cc2700 1 --2- 172.21.15.7:0/399532624 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ee007be30 0x7f6ee019bd30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:43:59.767 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.292+0000 7f6ee4cc2700 1 -- 172.21.15.7:0/399532624 >> 172.21.15.7:0/399532624 conn(0x7f6ee0101440 msgr2=0x7f6ee0102b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:43:59.767 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.292+0000 7f6ee4cc2700 1 -- 172.21.15.7:0/399532624 shutdown_connections 2023-07-20T02:43:59.767 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:43:59.292+0000 7f6ee4cc2700 1 -- 172.21.15.7:0/399532624 wait complete. 2023-07-20T02:43:59.767 INFO:teuthology.orchestra.run.smithi007.stdout:Restarting the monitor... 2023-07-20T02:44:00.192 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:43:59 smithi007 systemd[1]: Stopping Ceph mon.a for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:44:00.556 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:00 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mon-a[125821]: 2023-07-20T02:44:00.234+0000 7f9cd0b44700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:44:00.556 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:00 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mon-a[125821]: 2023-07-20T02:44:00.234+0000 7f9cd0b44700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-07-20T02:44:00.942 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:00 smithi007 bash[126180]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mon-a 2023-07-20T02:44:01.442 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 systemd[1]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mon.a.service: Succeeded. 2023-07-20T02:44:01.442 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 systemd[1]: Stopped Ceph mon.a for 1b04643c-26a7-11ee-9b34-001a4aab830c. 2023-07-20T02:44:01.443 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 systemd[1]: Starting Ceph mon.a for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:44:01.764 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 podman[126313]: 2023-07-20T02:44:01.764 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 bash[126313]: dd6dbd069f049b9b05c7785e9ee8935e176b75699aed79b8a5fff94c09f2a7e0 2023-07-20T02:44:01.764 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: set uid:gid to 167:167 (ceph:ceph) 2023-07-20T02:44:01.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev), process ceph-mon, pid 7 2023-07-20T02:44:01.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: pidfile_write: ignore empty --pid-file 2023-07-20T02:44:01.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: load: jerasure load: lrc 2023-07-20T02:44:01.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: RocksDB version: 7.9.2 2023-07-20T02:44:01.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Git sha 0 2023-07-20T02:44:01.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Compile date 2023-07-19 21:36:35 2023-07-20T02:44:01.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: DB SUMMARY 2023-07-20T02:44:01.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: DB Session ID: W6M0DKHLD0ZAH2421LGI 2023-07-20T02:44:01.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: CURRENT file: CURRENT 2023-07-20T02:44:01.836 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: IDENTITY file: IDENTITY 2023-07-20T02:44:01.836 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2023-07-20T02:44:01.836 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2023-07-20T02:44:01.836 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 97682 ; 2023-07-20T02:44:01.836 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.error_if_exists: 0 2023-07-20T02:44:01.837 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.create_if_missing: 0 2023-07-20T02:44:01.837 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.paranoid_checks: 1 2023-07-20T02:44:01.837 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-20T02:44:01.837 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-20T02:44:01.838 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-20T02:44:01.838 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.env: 0x560b457dfc60 2023-07-20T02:44:01.838 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.fs: PosixFileSystem 2023-07-20T02:44:01.838 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.info_log: 0x560b469bb2c0 2023-07-20T02:44:01.839 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.max_file_opening_threads: 16 2023-07-20T02:44:01.839 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.statistics: (nil) 2023-07-20T02:44:01.839 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.use_fsync: 0 2023-07-20T02:44:01.839 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.max_log_file_size: 0 2023-07-20T02:44:01.840 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-20T02:44:01.840 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-20T02:44:01.840 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.keep_log_file_num: 1000 2023-07-20T02:44:01.840 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.recycle_log_file_num: 0 2023-07-20T02:44:01.841 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.allow_fallocate: 1 2023-07-20T02:44:01.841 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.allow_mmap_reads: 0 2023-07-20T02:44:01.841 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.allow_mmap_writes: 0 2023-07-20T02:44:01.842 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.use_direct_reads: 0 2023-07-20T02:44:01.842 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-20T02:44:01.842 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.create_missing_column_families: 0 2023-07-20T02:44:01.843 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.db_log_dir: 2023-07-20T02:44:01.843 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.wal_dir: 2023-07-20T02:44:01.843 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-20T02:44:01.843 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-20T02:44:01.844 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-20T02:44:01.844 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-20T02:44:01.844 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-20T02:44:01.845 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-20T02:44:01.845 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.advise_random_on_open: 1 2023-07-20T02:44:01.845 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.db_write_buffer_size: 0 2023-07-20T02:44:01.845 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.write_buffer_manager: 0x560b468dc3c0 2023-07-20T02:44:01.846 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-20T02:44:01.846 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-20T02:44:01.846 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-20T02:44:01.846 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.rate_limiter: (nil) 2023-07-20T02:44:01.847 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-20T02:44:01.847 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.wal_recovery_mode: 2 2023-07-20T02:44:01.847 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.enable_thread_tracking: 0 2023-07-20T02:44:01.847 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.enable_pipelined_write: 0 2023-07-20T02:44:01.847 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.unordered_write: 0 2023-07-20T02:44:01.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-20T02:44:01.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-20T02:44:01.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-20T02:44:01.849 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-20T02:44:01.849 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.row_cache: None 2023-07-20T02:44:01.849 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.wal_filter: None 2023-07-20T02:44:01.849 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-20T02:44:01.850 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.allow_ingest_behind: 0 2023-07-20T02:44:01.850 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.two_write_queues: 0 2023-07-20T02:44:01.850 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.manual_wal_flush: 0 2023-07-20T02:44:01.850 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.wal_compression: 0 2023-07-20T02:44:01.851 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.atomic_flush: 0 2023-07-20T02:44:01.851 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-20T02:44:01.851 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-20T02:44:01.851 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-20T02:44:01.852 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.log_readahead_size: 0 2023-07-20T02:44:01.852 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-20T02:44:01.852 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.best_efforts_recovery: 0 2023-07-20T02:44:01.852 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-20T02:44:01.853 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-20T02:44:01.853 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.allow_data_in_errors: 0 2023-07-20T02:44:01.853 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.db_host_id: __hostname__ 2023-07-20T02:44:01.853 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-20T02:44:01.855 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.max_background_jobs: 2 2023-07-20T02:44:01.855 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.max_background_compactions: -1 2023-07-20T02:44:01.856 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.max_subcompactions: 1 2023-07-20T02:44:01.856 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-20T02:44:01.856 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-20T02:44:01.856 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-20T02:44:01.857 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.max_total_wal_size: 0 2023-07-20T02:44:01.857 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-20T02:44:01.857 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-20T02:44:01.857 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-20T02:44:01.858 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-20T02:44:01.858 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.max_open_files: -1 2023-07-20T02:44:01.858 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.bytes_per_sync: 0 2023-07-20T02:44:01.858 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-20T02:44:01.859 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-20T02:44:01.859 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.compaction_readahead_size: 0 2023-07-20T02:44:01.859 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.max_background_flushes: -1 2023-07-20T02:44:01.859 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Compression algorithms supported: 2023-07-20T02:44:01.860 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-20T02:44:01.860 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: kZSTD supported: 0 2023-07-20T02:44:01.860 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: kXpressCompression supported: 0 2023-07-20T02:44:01.860 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: kLZ4HCCompression supported: 1 2023-07-20T02:44:01.860 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: kZlibCompression supported: 1 2023-07-20T02:44:01.860 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: kSnappyCompression supported: 1 2023-07-20T02:44:01.861 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: kLZ4Compression supported: 1 2023-07-20T02:44:01.861 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: kBZip2Compression supported: 0 2023-07-20T02:44:01.861 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-20T02:44:01.861 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-20T02:44:01.861 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2023-07-20T02:44:01.861 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-20T02:44:01.861 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-20T02:44:01.862 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.merge_operator: 2023-07-20T02:44:01.862 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.compaction_filter: None 2023-07-20T02:44:01.862 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.compaction_filter_factory: None 2023-07-20T02:44:01.862 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.sst_partitioner_factory: None 2023-07-20T02:44:01.862 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-20T02:44:01.862 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-20T02:44:01.862 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560b469bb3c0) 2023-07-20T02:44:01.863 INFO:journalctl@ceph.mon.a.smithi007.stdout: cache_index_and_filter_blocks: 1 2023-07-20T02:44:01.863 INFO:journalctl@ceph.mon.a.smithi007.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-20T02:44:01.863 INFO:journalctl@ceph.mon.a.smithi007.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-20T02:44:01.863 INFO:journalctl@ceph.mon.a.smithi007.stdout: pin_top_level_index_and_filter: 1 2023-07-20T02:44:01.863 INFO:journalctl@ceph.mon.a.smithi007.stdout: index_type: 0 2023-07-20T02:44:01.863 INFO:journalctl@ceph.mon.a.smithi007.stdout: data_block_index_type: 0 2023-07-20T02:44:01.863 INFO:journalctl@ceph.mon.a.smithi007.stdout: index_shortening: 1 2023-07-20T02:44:01.864 INFO:journalctl@ceph.mon.a.smithi007.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-20T02:44:01.864 INFO:journalctl@ceph.mon.a.smithi007.stdout: checksum: 4 2023-07-20T02:44:01.864 INFO:journalctl@ceph.mon.a.smithi007.stdout: no_block_cache: 0 2023-07-20T02:44:01.864 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache: 0x560b4697b1f0 2023-07-20T02:44:01.864 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache_name: BinnedLRUCache 2023-07-20T02:44:01.864 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache_options: 2023-07-20T02:44:01.864 INFO:journalctl@ceph.mon.a.smithi007.stdout: capacity : 536870912 2023-07-20T02:44:01.865 INFO:journalctl@ceph.mon.a.smithi007.stdout: num_shard_bits : 4 2023-07-20T02:44:01.865 INFO:journalctl@ceph.mon.a.smithi007.stdout: strict_capacity_limit : 0 2023-07-20T02:44:01.865 INFO:journalctl@ceph.mon.a.smithi007.stdout: high_pri_pool_ratio: 0.000 2023-07-20T02:44:01.865 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache_compressed: (nil) 2023-07-20T02:44:01.865 INFO:journalctl@ceph.mon.a.smithi007.stdout: persistent_cache: (nil) 2023-07-20T02:44:01.865 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_size: 4096 2023-07-20T02:44:01.865 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_size_deviation: 10 2023-07-20T02:44:01.866 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_restart_interval: 16 2023-07-20T02:44:01.866 INFO:journalctl@ceph.mon.a.smithi007.stdout: index_block_restart_interval: 1 2023-07-20T02:44:01.866 INFO:journalctl@ceph.mon.a.smithi007.stdout: metadata_block_size: 4096 2023-07-20T02:44:01.866 INFO:journalctl@ceph.mon.a.smithi007.stdout: partition_filters: 0 2023-07-20T02:44:01.866 INFO:journalctl@ceph.mon.a.smithi007.stdout: use_delta_encoding: 1 2023-07-20T02:44:01.866 INFO:journalctl@ceph.mon.a.smithi007.stdout: filter_policy: bloomfilter 2023-07-20T02:44:01.866 INFO:journalctl@ceph.mon.a.smithi007.stdout: whole_key_filtering: 1 2023-07-20T02:44:01.866 INFO:journalctl@ceph.mon.a.smithi007.stdout: verify_compression: 0 2023-07-20T02:44:01.867 INFO:journalctl@ceph.mon.a.smithi007.stdout: read_amp_bytes_per_bit: 0 2023-07-20T02:44:01.867 INFO:journalctl@ceph.mon.a.smithi007.stdout: format_version: 5 2023-07-20T02:44:01.867 INFO:journalctl@ceph.mon.a.smithi007.stdout: enable_index_compression: 1 2023-07-20T02:44:01.867 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_align: 0 2023-07-20T02:44:01.867 INFO:journalctl@ceph.mon.a.smithi007.stdout: max_auto_readahead_size: 262144 2023-07-20T02:44:01.867 INFO:journalctl@ceph.mon.a.smithi007.stdout: prepopulate_block_cache: 0 2023-07-20T02:44:01.867 INFO:journalctl@ceph.mon.a.smithi007.stdout: initial_auto_readahead_size: 8192 2023-07-20T02:44:01.868 INFO:journalctl@ceph.mon.a.smithi007.stdout: num_file_reads_for_auto_readahead: 2 2023-07-20T02:44:01.868 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.write_buffer_size: 33554432 2023-07-20T02:44:01.868 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.max_write_buffer_number: 2 2023-07-20T02:44:01.868 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.compression: NoCompression 2023-07-20T02:44:01.868 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.bottommost_compression: Disabled 2023-07-20T02:44:01.868 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.prefix_extractor: nullptr 2023-07-20T02:44:01.868 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-20T02:44:01.869 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.num_levels: 7 2023-07-20T02:44:01.869 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-20T02:44:01.869 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-20T02:44:01.869 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-20T02:44:01.869 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-20T02:44:01.869 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-20T02:44:01.869 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-20T02:44:01.870 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-20T02:44:01.870 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-20T02:44:01.870 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-20T02:44:01.870 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-20T02:44:01.870 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-20T02:44:01.870 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-20T02:44:01.870 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-20T02:44:01.871 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.compression_opts.level: 32767 2023-07-20T02:44:01.871 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.compression_opts.strategy: 0 2023-07-20T02:44:01.871 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-20T02:44:01.871 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-20T02:44:01.871 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-20T02:44:01.871 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-20T02:44:01.871 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.compression_opts.enabled: false 2023-07-20T02:44:01.872 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-20T02:44:01.872 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-20T02:44:01.872 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-20T02:44:01.872 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-20T02:44:01.872 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.target_file_size_base: 67108864 2023-07-20T02:44:01.872 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-20T02:44:01.872 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-20T02:44:01.873 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-20T02:44:01.873 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-20T02:44:01.873 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-20T02:44:01.873 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-20T02:44:01.873 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-20T02:44:01.873 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-20T02:44:01.873 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-20T02:44:01.874 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-20T02:44:01.874 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-20T02:44:01.874 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-20T02:44:01.874 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-20T02:44:01.874 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-20T02:44:01.874 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.arena_block_size: 1048576 2023-07-20T02:44:01.874 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-20T02:44:01.875 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-20T02:44:01.875 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.disable_auto_compactions: 0 2023-07-20T02:44:01.875 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-20T02:44:01.875 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-20T02:44:01.875 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-20T02:44:01.875 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-20T02:44:01.875 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-20T02:44:01.876 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-20T02:44:01.876 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-20T02:44:01.876 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-20T02:44:01.876 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-20T02:44:01.876 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-20T02:44:01.876 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.table_properties_collectors: 2023-07-20T02:44:01.876 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.inplace_update_support: 0 2023-07-20T02:44:01.877 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-20T02:44:01.877 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-20T02:44:01.877 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-20T02:44:01.877 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-20T02:44:01.877 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.bloom_locality: 0 2023-07-20T02:44:01.877 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.max_successive_merges: 0 2023-07-20T02:44:01.877 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-20T02:44:01.878 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.paranoid_file_checks: 0 2023-07-20T02:44:01.878 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.force_consistency_checks: 1 2023-07-20T02:44:01.878 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.report_bg_io_stats: 0 2023-07-20T02:44:01.878 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.ttl: 2592000 2023-07-20T02:44:01.878 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-20T02:44:01.878 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-20T02:44:01.878 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-20T02:44:01.879 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.enable_blob_files: false 2023-07-20T02:44:01.879 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.min_blob_size: 0 2023-07-20T02:44:01.879 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.blob_file_size: 268435456 2023-07-20T02:44:01.879 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-20T02:44:01.879 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-20T02:44:01.879 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-20T02:44:01.879 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-20T02:44:01.879 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-20T02:44:01.880 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.blob_file_starting_level: 0 2023-07-20T02:44:01.880 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-20T02:44:01.880 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2023-07-20T02:44:01.880 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2023-07-20T02:44:01.880 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 978befee-d814-4932-9f5b-1b2ff156cf07 2023-07-20T02:44:01.880 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689821041765989, "job": 1, "event": "recovery_started", "wal_files": [9]} 2023-07-20T02:44:01.881 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2023-07-20T02:44:01.881 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689821041767728, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 92991, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 310, "table_properties": {"data_size": 91013, "index_size": 260, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13398, "raw_average_key_size": 49, "raw_value_size": 83722, "raw_average_value_size": 311, "num_data_blocks": 11, "num_entries": 269, "num_filter_entries": 269, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689821041, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "978befee-d814-4932-9f5b-1b2ff156cf07", "db_session_id": "W6M0DKHLD0ZAH2421LGI", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2023-07-20T02:44:01.881 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689821041767984, "job": 1, "event": "recovery_finished"} 2023-07-20T02:44:01.881 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2023-07-20T02:44:01.881 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-20T02:44:01.881 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x560b46a14000 2023-07-20T02:44:01.881 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: DB pointer 0x560b46a00000 2023-07-20T02:44:01.882 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-20T02:44:01.882 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-20T02:44:01.882 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** DB Stats ** 2023-07-20T02:44:01.882 INFO:journalctl@ceph.mon.a.smithi007.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T02:44:01.882 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-20T02:44:01.882 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T02:44:01.882 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T02:44:01.883 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-20T02:44:01.883 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T02:44:01.883 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T02:44:01.883 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2023-07-20T02:44:01.883 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** Compaction Stats [default] ** 2023-07-20T02:44:01.883 INFO:journalctl@ceph.mon.a.smithi007.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T02:44:01.883 INFO:journalctl@ceph.mon.a.smithi007.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-20T02:44:01.884 INFO:journalctl@ceph.mon.a.smithi007.stdout: L0 2/0 92.65 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 77.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T02:44:01.884 INFO:journalctl@ceph.mon.a.smithi007.stdout: Sum 2/0 92.65 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 77.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T02:44:01.884 INFO:journalctl@ceph.mon.a.smithi007.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 77.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T02:44:01.884 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2023-07-20T02:44:01.884 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** Compaction Stats [default] ** 2023-07-20T02:44:01.884 INFO:journalctl@ceph.mon.a.smithi007.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T02:44:01.884 INFO:journalctl@ceph.mon.a.smithi007.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-20T02:44:01.885 INFO:journalctl@ceph.mon.a.smithi007.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 77.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T02:44:01.885 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2023-07-20T02:44:01.885 INFO:journalctl@ceph.mon.a.smithi007.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-20T02:44:01.885 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2023-07-20T02:44:01.885 INFO:journalctl@ceph.mon.a.smithi007.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T02:44:01.885 INFO:journalctl@ceph.mon.a.smithi007.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-20T02:44:01.885 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-20T02:44:01.886 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-20T02:44:01.886 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-20T02:44:01.886 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-20T02:44:01.886 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative compaction: 0.00 GB write, 14.16 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T02:44:01.886 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval compaction: 0.00 GB write, 14.16 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T02:44:01.886 INFO:journalctl@ceph.mon.a.smithi007.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-20T02:44:01.886 INFO:journalctl@ceph.mon.a.smithi007.stdout: Block cache BinnedLRUCache@0x560b4697b1f0#7 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2023-07-20T02:44:01.887 INFO:journalctl@ceph.mon.a.smithi007.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2023-07-20T02:44:01.887 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2023-07-20T02:44:01.887 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-20T02:44:01.887 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: starting mon.a rank 0 at public addrs [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] at bind addrs [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 1b04643c-26a7-11ee-9b34-001a4aab830c 2023-07-20T02:44:01.887 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: mon.a@-1(???) e1 preinit fsid 1b04643c-26a7-11ee-9b34-001a4aab830c 2023-07-20T02:44:01.887 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: mon.a@-1(???).mds e1 new map 2023-07-20T02:44:01.887 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: mon.a@-1(???).mds e1 print_map 2023-07-20T02:44:01.888 INFO:journalctl@ceph.mon.a.smithi007.stdout: e1 2023-07-20T02:44:01.888 INFO:journalctl@ceph.mon.a.smithi007.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-20T02:44:01.888 INFO:journalctl@ceph.mon.a.smithi007.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-20T02:44:01.888 INFO:journalctl@ceph.mon.a.smithi007.stdout: legacy client fscid: -1 2023-07-20T02:44:01.888 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2023-07-20T02:44:01.888 INFO:journalctl@ceph.mon.a.smithi007.stdout: No filesystems configured 2023-07-20T02:44:01.888 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-20T02:44:01.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T02:44:01.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T02:44:01.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T02:44:01.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-07-20T02:44:01.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: expand_channel_meta expand map: {default=false} 2023-07-20T02:44:01.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: expand_channel_meta from 'false' to 'false' 2023-07-20T02:44:01.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: expand_channel_meta expanded map: {default=false} 2023-07-20T02:44:01.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: expand_channel_meta expand map: {default=info} 2023-07-20T02:44:01.890 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: expand_channel_meta from 'info' to 'info' 2023-07-20T02:44:01.890 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: expand_channel_meta expanded map: {default=info} 2023-07-20T02:44:01.890 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: expand_channel_meta expand map: {default=daemon} 2023-07-20T02:44:01.890 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-20T02:44:01.890 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: expand_channel_meta expanded map: {default=daemon} 2023-07-20T02:44:01.890 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: expand_channel_meta expand map: {default=debug} 2023-07-20T02:44:01.890 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: expand_channel_meta from 'debug' to 'debug' 2023-07-20T02:44:01.891 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: expand_channel_meta expanded map: {default=debug} 2023-07-20T02:44:01.891 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 systemd[1]: Started Ceph mon.a for 1b04643c-26a7-11ee-9b34-001a4aab830c. 2023-07-20T02:44:01.891 INFO:teuthology.orchestra.run.smithi007.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2023-07-20T02:44:02.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-20T02:44:02.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: monmap e1: 1 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0]} removed_ranks: {} 2023-07-20T02:44:02.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: fsmap 2023-07-20T02:44:02.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: osdmap e1: 0 total, 0 up, 0 in 2023-07-20T02:44:02.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:01 smithi007 ceph-mon[126353]: mgrmap e1: no daemons active 2023-07-20T02:44:02.924 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.339+0000 7f0371d77700 1 Processor -- start 2023-07-20T02:44:02.925 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.339+0000 7f0371d77700 1 -- start start 2023-07-20T02:44:02.925 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.339+0000 7f0371d77700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f036c07be70 0x7f036c07a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:02.925 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.339+0000 7f0371d77700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036c07a8a0 con 0x7f036c07be70 2023-07-20T02:44:02.925 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.340+0000 7f036b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f036c07be70 0x7f036c07a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:02.925 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.340+0000 7f036b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f036c07be70 0x7f036c07a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54600/0 (socket says 172.21.15.7:54600) 2023-07-20T02:44:02.926 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.340+0000 7f036b7fe700 1 -- 172.21.15.7:0/401722229 learned_addr learned my addr 172.21.15.7:0/401722229 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:02.926 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.340+0000 7f036b7fe700 1 -- 172.21.15.7:0/401722229 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f036c07a9e0 con 0x7f036c07be70 2023-07-20T02:44:02.926 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.341+0000 7f036b7fe700 1 --2- 172.21.15.7:0/401722229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f036c07be70 0x7f036c07a2d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f035c0098a0 tx=0x7f035c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bd9e8408ed88f8d server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:02.926 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.341+0000 7f036a7fc700 1 -- 172.21.15.7:0/401722229 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f035c004160 con 0x7f036c07be70 2023-07-20T02:44:02.926 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.341+0000 7f036a7fc700 1 -- 172.21.15.7:0/401722229 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f035c0042c0 con 0x7f036c07be70 2023-07-20T02:44:02.926 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.341+0000 7f036a7fc700 1 -- 172.21.15.7:0/401722229 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f035c011a00 con 0x7f036c07be70 2023-07-20T02:44:02.926 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.341+0000 7f0371d77700 1 -- 172.21.15.7:0/401722229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f036c07be70 msgr2=0x7f036c07a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:02.926 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.341+0000 7f0371d77700 1 --2- 172.21.15.7:0/401722229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f036c07be70 0x7f036c07a2d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f035c0098a0 tx=0x7f035c009bb0 comp rx=0 tx=0).stop 2023-07-20T02:44:02.927 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.342+0000 7f0371d77700 1 -- 172.21.15.7:0/401722229 shutdown_connections 2023-07-20T02:44:02.927 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.342+0000 7f0371d77700 1 --2- 172.21.15.7:0/401722229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f036c07be70 0x7f036c07a2d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:02.927 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.342+0000 7f0371d77700 1 -- 172.21.15.7:0/401722229 >> 172.21.15.7:0/401722229 conn(0x7f036c101440 msgr2=0x7f036c103860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:02.927 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.342+0000 7f0371d77700 1 -- 172.21.15.7:0/401722229 shutdown_connections 2023-07-20T02:44:02.927 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.342+0000 7f0371d77700 1 -- 172.21.15.7:0/401722229 wait complete. 2023-07-20T02:44:02.927 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.343+0000 7f0371d77700 1 Processor -- start 2023-07-20T02:44:02.927 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.343+0000 7f0371d77700 1 -- start start 2023-07-20T02:44:02.928 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.343+0000 7f0371d77700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f036c07be70 0x7f036c19bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:02.928 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.343+0000 7f0371d77700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036c19c2f0 con 0x7f036c07be70 2023-07-20T02:44:02.928 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.343+0000 7f036b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f036c07be70 0x7f036c19bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:02.928 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.344+0000 7f036b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f036c07be70 0x7f036c19bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54608/0 (socket says 172.21.15.7:54608) 2023-07-20T02:44:02.928 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.344+0000 7f036b7fe700 1 -- 172.21.15.7:0/1070360203 learned_addr learned my addr 172.21.15.7:0/1070360203 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:02.928 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.344+0000 7f036b7fe700 1 -- 172.21.15.7:0/1070360203 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f035c009550 con 0x7f036c07be70 2023-07-20T02:44:02.928 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.344+0000 7f036b7fe700 1 --2- 172.21.15.7:0/1070360203 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f036c07be70 0x7f036c19bdb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f035c006af0 tx=0x7f035c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:02.928 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.344+0000 7f0368ff9700 1 -- 172.21.15.7:0/1070360203 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f035c003a40 con 0x7f036c07be70 2023-07-20T02:44:02.929 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.344+0000 7f0368ff9700 1 -- 172.21.15.7:0/1070360203 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f035c003ba0 con 0x7f036c07be70 2023-07-20T02:44:02.929 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.344+0000 7f0371d77700 1 -- 172.21.15.7:0/1070360203 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f036c19c4f0 con 0x7f036c07be70 2023-07-20T02:44:02.929 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.344+0000 7f0368ff9700 1 -- 172.21.15.7:0/1070360203 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f035c01a700 con 0x7f036c07be70 2023-07-20T02:44:02.929 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.345+0000 7f0371d77700 1 -- 172.21.15.7:0/1070360203 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f036c19c910 con 0x7f036c07be70 2023-07-20T02:44:02.929 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.345+0000 7f0368ff9700 1 -- 172.21.15.7:0/1070360203 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f035c01ac80 con 0x7f036c07be70 2023-07-20T02:44:02.929 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.345+0000 7f0368ff9700 1 -- 172.21.15.7:0/1070360203 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f035c019700 con 0x7f036c07be70 2023-07-20T02:44:02.929 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.346+0000 7f0371d77700 1 -- 172.21.15.7:0/1070360203 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f036c195e40 con 0x7f036c07be70 2023-07-20T02:44:02.930 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.348+0000 7f0368ff9700 1 -- 172.21.15.7:0/1070360203 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f035c024ad0 con 0x7f036c07be70 2023-07-20T02:44:02.930 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.421+0000 7f0371d77700 1 -- 172.21.15.7:0/1070360203 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f036c02c970 con 0x7f036c07be70 2023-07-20T02:44:02.930 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.424+0000 7f0368ff9700 1 -- 172.21.15.7:0/1070360203 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f035c024c70 con 0x7f036c07be70 2023-07-20T02:44:02.930 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.424+0000 7f0371d77700 1 -- 172.21.15.7:0/1070360203 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f036c07be70 msgr2=0x7f036c19bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:02.930 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.424+0000 7f0371d77700 1 --2- 172.21.15.7:0/1070360203 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f036c07be70 0x7f036c19bdb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f035c006af0 tx=0x7f035c011ad0 comp rx=0 tx=0).stop 2023-07-20T02:44:02.930 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.425+0000 7f0371d77700 1 -- 172.21.15.7:0/1070360203 shutdown_connections 2023-07-20T02:44:02.930 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.425+0000 7f0371d77700 1 --2- 172.21.15.7:0/1070360203 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f036c07be70 0x7f036c19bdb0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:02.930 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.425+0000 7f0371d77700 1 -- 172.21.15.7:0/1070360203 >> 172.21.15.7:0/1070360203 conn(0x7f036c101440 msgr2=0x7f036c102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:02.931 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.425+0000 7f0371d77700 1 -- 172.21.15.7:0/1070360203 shutdown_connections 2023-07-20T02:44:02.931 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:02.425+0000 7f0371d77700 1 -- 172.21.15.7:0/1070360203 wait complete. 2023-07-20T02:44:02.931 INFO:teuthology.orchestra.run.smithi007.stdout:Wrote config to /etc/ceph/ceph.conf 2023-07-20T02:44:02.950 INFO:teuthology.orchestra.run.smithi007.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-07-20T02:44:02.951 INFO:teuthology.orchestra.run.smithi007.stdout:Creating mgr... 2023-07-20T02:44:02.951 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying port 9283 ... 2023-07-20T02:44:02.951 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying port 8765 ... 2023-07-20T02:44:03.327 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mgr.x 2023-07-20T02:44:03.328 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Failed to reset failed state of unit ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mgr.x.service: Unit ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mgr.x.service not loaded. 2023-07-20T02:44:03.530 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-1b04643c-26a7-11ee-9b34-001a4aab830c.target.wants/ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mgr.x.service → /etc/systemd/system/ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@.service. 2023-07-20T02:44:03.693 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:03 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/1070360203' entity='client.admin' 2023-07-20T02:44:03.694 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:03 smithi007 systemd[1]: Starting Ceph mgr.x for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:44:04.318 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:04 smithi007 podman[126583]: 2023-07-20T02:44:04.318 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:04 smithi007 bash[126583]: 592ba3a20e5c0b48219f0a438d6a6df99ea7a488053bd8c602a9585e5bf951e0 2023-07-20T02:44:04.343 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-20T02:44:04.344 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout disabled 2023-07-20T02:44:04.360 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-20T02:44:04.360 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2023-07-20T02:44:04.360 INFO:teuthology.orchestra.run.smithi007.stdout:firewalld.service is not enabled 2023-07-20T02:44:04.361 INFO:teuthology.orchestra.run.smithi007.stdout:Not possible to enable service . firewalld.service is not available 2023-07-20T02:44:04.378 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-20T02:44:04.379 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout disabled 2023-07-20T02:44:04.395 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-20T02:44:04.395 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2023-07-20T02:44:04.395 INFO:teuthology.orchestra.run.smithi007.stdout:firewalld.service is not enabled 2023-07-20T02:44:04.395 INFO:teuthology.orchestra.run.smithi007.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2023-07-20T02:44:04.395 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mgr to start... 2023-07-20T02:44:04.396 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mgr... 2023-07-20T02:44:04.692 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:04 smithi007 systemd[1]: Started Ceph mgr.x for 1b04643c-26a7-11ee-9b34-001a4aab830c. 2023-07-20T02:44:04.692 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:04 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:04.450+0000 7f2db8170200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T02:44:04.692 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:04 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:04.560+0000 7f2db8170200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T02:44:04.693 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:04 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:04.684+0000 7f2db8170200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T02:44:04.942 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:04 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:04.814+0000 7f2db8170200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T02:44:05.442 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:05 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/597235874' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-20T02:44:05.442 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:05 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:05.051+0000 7f2db8170200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T02:44:05.493 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2023-07-20T02:44:05.493 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2023-07-20T02:44:05.493 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsid": "1b04643c-26a7-11ee-9b34-001a4aab830c", 2023-07-20T02:44:05.494 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "health": { 2023-07-20T02:44:05.494 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-20T02:44:05.494 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-20T02:44:05.494 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-20T02:44:05.494 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:44:05.495 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-20T02:44:05.495 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-20T02:44:05.495 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 0 2023-07-20T02:44:05.495 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2023-07-20T02:44:05.496 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-20T02:44:05.496 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "a" 2023-07-20T02:44:05.496 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2023-07-20T02:44:05.497 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2023-07-20T02:44:05.497 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-20T02:44:05.497 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T02:44:05.497 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-20T02:44:05.497 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-20T02:44:05.498 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:44:05.498 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-20T02:44:05.498 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T02:44:05.498 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-20T02:44:05.498 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-20T02:44:05.498 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-20T02:44:05.498 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-20T02:44:05.498 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-20T02:44:05.499 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-20T02:44:05.499 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:44:05.499 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-20T02:44:05.499 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-20T02:44:05.499 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-20T02:44:05.499 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-20T02:44:05.499 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-20T02:44:05.499 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-20T02:44:05.500 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-20T02:44:05.500 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-20T02:44:05.500 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-20T02:44:05.500 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:44:05.500 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-20T02:44:05.500 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T02:44:05.500 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-20T02:44:05.501 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-20T02:44:05.501 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:44:05.501 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-20T02:44:05.501 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-20T02:44:05.501 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-20T02:44:05.501 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-20T02:44:05.501 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-20T02:44:05.502 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-20T02:44:05.502 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "restful" 2023-07-20T02:44:05.502 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2023-07-20T02:44:05.502 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T02:44:05.502 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:44:05.503 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-20T02:44:05.503 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T02:44:05.503 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modified": "2023-07-20T02:43:56.545172+0000", 2023-07-20T02:44:05.503 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T02:44:05.503 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:44:05.503 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-20T02:44:05.504 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2023-07-20T02:44:05.504 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.969+0000 7fccd3a5a700 1 Processor -- start 2023-07-20T02:44:05.504 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.969+0000 7fccd3a5a700 1 -- start start 2023-07-20T02:44:05.504 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.970+0000 7fccd3a5a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcccc07be50 0x7fcccc07a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:05.504 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.970+0000 7fccd3a5a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc07a880 con 0x7fcccc07be50 2023-07-20T02:44:05.504 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.970+0000 7fccd17f6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcccc07be50 0x7fcccc07a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:05.505 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.970+0000 7fccd17f6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcccc07be50 0x7fcccc07a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54638/0 (socket says 172.21.15.7:54638) 2023-07-20T02:44:05.505 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.970+0000 7fccd17f6700 1 -- 172.21.15.7:0/3569252469 learned_addr learned my addr 172.21.15.7:0/3569252469 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:05.505 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.971+0000 7fccd17f6700 1 -- 172.21.15.7:0/3569252469 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcccc07a9c0 con 0x7fcccc07be50 2023-07-20T02:44:05.505 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.971+0000 7fccd17f6700 1 --2- 172.21.15.7:0/3569252469 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcccc07be50 0x7fcccc07a2b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fccc80098a0 tx=0x7fccc8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b066cac8977df997 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:05.506 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.971+0000 7fccc3fff700 1 -- 172.21.15.7:0/3569252469 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fccc8004160 con 0x7fcccc07be50 2023-07-20T02:44:05.506 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.971+0000 7fccc3fff700 1 -- 172.21.15.7:0/3569252469 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fccc80042c0 con 0x7fcccc07be50 2023-07-20T02:44:05.506 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.971+0000 7fccc3fff700 1 -- 172.21.15.7:0/3569252469 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fccc8011a00 con 0x7fcccc07be50 2023-07-20T02:44:05.506 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.972+0000 7fccd3a5a700 1 -- 172.21.15.7:0/3569252469 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcccc07be50 msgr2=0x7fcccc07a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:05.506 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.972+0000 7fccd3a5a700 1 --2- 172.21.15.7:0/3569252469 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcccc07be50 0x7fcccc07a2b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fccc80098a0 tx=0x7fccc8009bb0 comp rx=0 tx=0).stop 2023-07-20T02:44:05.506 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.972+0000 7fccd3a5a700 1 -- 172.21.15.7:0/3569252469 shutdown_connections 2023-07-20T02:44:05.506 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.972+0000 7fccd3a5a700 1 --2- 172.21.15.7:0/3569252469 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcccc07be50 0x7fcccc07a2b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:05.507 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.972+0000 7fccd3a5a700 1 -- 172.21.15.7:0/3569252469 >> 172.21.15.7:0/3569252469 conn(0x7fcccc101440 msgr2=0x7fcccc103860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:05.507 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.972+0000 7fccd3a5a700 1 -- 172.21.15.7:0/3569252469 shutdown_connections 2023-07-20T02:44:05.507 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.972+0000 7fccd3a5a700 1 -- 172.21.15.7:0/3569252469 wait complete. 2023-07-20T02:44:05.507 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.974+0000 7fccd3a5a700 1 Processor -- start 2023-07-20T02:44:05.507 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.974+0000 7fccd3a5a700 1 -- start start 2023-07-20T02:44:05.507 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.974+0000 7fccd3a5a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcccc07be50 0x7fcccc19bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:05.507 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.974+0000 7fccd3a5a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc19c270 con 0x7fcccc07be50 2023-07-20T02:44:05.508 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.975+0000 7fccd17f6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcccc07be50 0x7fcccc19bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:05.508 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.975+0000 7fccd17f6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcccc07be50 0x7fcccc19bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54642/0 (socket says 172.21.15.7:54642) 2023-07-20T02:44:05.508 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.975+0000 7fccd17f6700 1 -- 172.21.15.7:0/597235874 learned_addr learned my addr 172.21.15.7:0/597235874 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:05.508 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.975+0000 7fccd17f6700 1 -- 172.21.15.7:0/597235874 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc8009550 con 0x7fcccc07be50 2023-07-20T02:44:05.509 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.975+0000 7fccd17f6700 1 --2- 172.21.15.7:0/597235874 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcccc07be50 0x7fcccc19bd30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fccc8006af0 tx=0x7fccc8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:05.509 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.975+0000 7fccc27fc700 1 -- 172.21.15.7:0/597235874 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fccc8003a40 con 0x7fcccc07be50 2023-07-20T02:44:05.509 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.976+0000 7fccd3a5a700 1 -- 172.21.15.7:0/597235874 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcccc19c470 con 0x7fcccc07be50 2023-07-20T02:44:05.509 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.976+0000 7fccc27fc700 1 -- 172.21.15.7:0/597235874 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fccc8003ba0 con 0x7fcccc07be50 2023-07-20T02:44:05.509 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.976+0000 7fccc27fc700 1 -- 172.21.15.7:0/597235874 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fccc801a6e0 con 0x7fcccc07be50 2023-07-20T02:44:05.510 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.976+0000 7fccd3a5a700 1 -- 172.21.15.7:0/597235874 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcccc19c890 con 0x7fcccc07be50 2023-07-20T02:44:05.510 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.976+0000 7fccc27fc700 1 -- 172.21.15.7:0/597235874 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fccc801ac50 con 0x7fcccc07be50 2023-07-20T02:44:05.510 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.976+0000 7fccc27fc700 1 -- 172.21.15.7:0/597235874 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fccc8019700 con 0x7fcccc07be50 2023-07-20T02:44:05.510 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.977+0000 7fccd3a5a700 1 -- 172.21.15.7:0/597235874 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcccc10c250 con 0x7fcccc07be50 2023-07-20T02:44:05.510 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:04.980+0000 7fccc27fc700 1 -- 172.21.15.7:0/597235874 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fccc802de40 con 0x7fcccc07be50 2023-07-20T02:44:05.510 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:05.057+0000 7fccd3a5a700 1 -- 172.21.15.7:0/597235874 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fcccc19f8a0 con 0x7fcccc07be50 2023-07-20T02:44:05.510 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:05.057+0000 7fccc27fc700 1 -- 172.21.15.7:0/597235874 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fccc8024a60 con 0x7fcccc07be50 2023-07-20T02:44:05.511 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:05.058+0000 7fccd3a5a700 1 -- 172.21.15.7:0/597235874 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcccc07be50 msgr2=0x7fcccc19bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:05.511 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:05.058+0000 7fccd3a5a700 1 --2- 172.21.15.7:0/597235874 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcccc07be50 0x7fcccc19bd30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fccc8006af0 tx=0x7fccc8011ad0 comp rx=0 tx=0).stop 2023-07-20T02:44:05.511 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:05.058+0000 7fccd3a5a700 1 -- 172.21.15.7:0/597235874 shutdown_connections 2023-07-20T02:44:05.511 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:05.058+0000 7fccd3a5a700 1 --2- 172.21.15.7:0/597235874 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcccc07be50 0x7fcccc19bd30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:05.511 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:05.058+0000 7fccd3a5a700 1 -- 172.21.15.7:0/597235874 >> 172.21.15.7:0/597235874 conn(0x7fcccc101440 msgr2=0x7fcccc102b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:05.511 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:05.059+0000 7fccd3a5a700 1 -- 172.21.15.7:0/597235874 shutdown_connections 2023-07-20T02:44:05.511 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:05.059+0000 7fccd3a5a700 1 -- 172.21.15.7:0/597235874 wait complete. 2023-07-20T02:44:05.512 INFO:teuthology.orchestra.run.smithi007.stdout:mgr not available, waiting (1/15)... 2023-07-20T02:44:05.775 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:05 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:05.481+0000 7f2db8170200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T02:44:05.775 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:05 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:05.601+0000 7f2db8170200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T02:44:06.192 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:05 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:05.767+0000 7f2db8170200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T02:44:06.192 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:05 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:05.896+0000 7f2db8170200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T02:44:06.942 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:06 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:06.459+0000 7f2db8170200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T02:44:07.692 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:07 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:07.267+0000 7f2db8170200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T02:44:07.693 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:07 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:07.478+0000 7f2db8170200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T02:44:08.442 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:08 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/3395818950' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-20T02:44:08.571 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2023-07-20T02:44:08.571 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2023-07-20T02:44:08.571 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsid": "1b04643c-26a7-11ee-9b34-001a4aab830c", 2023-07-20T02:44:08.571 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "health": { 2023-07-20T02:44:08.571 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-20T02:44:08.572 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-20T02:44:08.572 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-20T02:44:08.572 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:44:08.572 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-20T02:44:08.572 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-20T02:44:08.572 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 0 2023-07-20T02:44:08.572 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2023-07-20T02:44:08.573 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-20T02:44:08.573 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "a" 2023-07-20T02:44:08.573 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2023-07-20T02:44:08.573 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2023-07-20T02:44:08.573 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-20T02:44:08.573 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T02:44:08.573 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-20T02:44:08.573 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-20T02:44:08.574 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:44:08.574 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-20T02:44:08.574 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T02:44:08.574 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-20T02:44:08.574 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-20T02:44:08.574 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-20T02:44:08.580 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-20T02:44:08.580 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-20T02:44:08.580 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-20T02:44:08.581 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:44:08.581 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-20T02:44:08.581 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-20T02:44:08.581 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-20T02:44:08.581 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-20T02:44:08.581 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-20T02:44:08.581 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-20T02:44:08.582 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-20T02:44:08.582 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-20T02:44:08.582 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-20T02:44:08.582 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:44:08.582 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-20T02:44:08.582 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T02:44:08.582 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-20T02:44:08.582 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-20T02:44:08.583 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:44:08.583 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-20T02:44:08.583 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-20T02:44:08.583 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-20T02:44:08.583 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-20T02:44:08.583 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-20T02:44:08.583 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-20T02:44:08.584 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "restful" 2023-07-20T02:44:08.584 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2023-07-20T02:44:08.584 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T02:44:08.584 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:44:08.584 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-20T02:44:08.584 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T02:44:08.584 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modified": "2023-07-20T02:43:56.545172+0000", 2023-07-20T02:44:08.584 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T02:44:08.585 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:44:08.585 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-20T02:44:08.585 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2023-07-20T02:44:08.585 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.032+0000 7fb4d014f700 1 Processor -- start 2023-07-20T02:44:08.585 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.032+0000 7fb4d014f700 1 -- start start 2023-07-20T02:44:08.585 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.033+0000 7fb4d014f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4c807be30 0x7fb4c807a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:08.585 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.033+0000 7fb4d014f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c807a860 con 0x7fb4c807be30 2023-07-20T02:44:08.586 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.033+0000 7fb4cdeeb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4c807be30 0x7fb4c807a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:08.586 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.033+0000 7fb4cdeeb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4c807be30 0x7fb4c807a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54658/0 (socket says 172.21.15.7:54658) 2023-07-20T02:44:08.586 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.033+0000 7fb4cdeeb700 1 -- 172.21.15.7:0/882123354 learned_addr learned my addr 172.21.15.7:0/882123354 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:08.586 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.034+0000 7fb4cdeeb700 1 -- 172.21.15.7:0/882123354 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c807a9a0 con 0x7fb4c807be30 2023-07-20T02:44:08.586 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.034+0000 7fb4cdeeb700 1 --2- 172.21.15.7:0/882123354 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4c807be30 0x7fb4c807a290 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb4c40098a0 tx=0x7fb4c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad026e6a4137edf server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:08.586 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.034+0000 7fb4ccee9700 1 -- 172.21.15.7:0/882123354 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4c4004160 con 0x7fb4c807be30 2023-07-20T02:44:08.586 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.034+0000 7fb4ccee9700 1 -- 172.21.15.7:0/882123354 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb4c40042c0 con 0x7fb4c807be30 2023-07-20T02:44:08.587 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.034+0000 7fb4ccee9700 1 -- 172.21.15.7:0/882123354 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4c4011a00 con 0x7fb4c807be30 2023-07-20T02:44:08.587 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.035+0000 7fb4d014f700 1 -- 172.21.15.7:0/882123354 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4c807be30 msgr2=0x7fb4c807a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:08.587 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.035+0000 7fb4d014f700 1 --2- 172.21.15.7:0/882123354 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4c807be30 0x7fb4c807a290 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb4c40098a0 tx=0x7fb4c4009bb0 comp rx=0 tx=0).stop 2023-07-20T02:44:08.587 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.035+0000 7fb4d014f700 1 -- 172.21.15.7:0/882123354 shutdown_connections 2023-07-20T02:44:08.587 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.035+0000 7fb4d014f700 1 --2- 172.21.15.7:0/882123354 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4c807be30 0x7fb4c807a290 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:08.587 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.035+0000 7fb4d014f700 1 -- 172.21.15.7:0/882123354 >> 172.21.15.7:0/882123354 conn(0x7fb4c8101440 msgr2=0x7fb4c8103860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:08.587 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.035+0000 7fb4d014f700 1 -- 172.21.15.7:0/882123354 shutdown_connections 2023-07-20T02:44:08.588 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.035+0000 7fb4d014f700 1 -- 172.21.15.7:0/882123354 wait complete. 2023-07-20T02:44:08.588 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.037+0000 7fb4d014f700 1 Processor -- start 2023-07-20T02:44:08.588 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.037+0000 7fb4d014f700 1 -- start start 2023-07-20T02:44:08.588 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.037+0000 7fb4d014f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4c807be30 0x7fb4c819bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:08.588 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.037+0000 7fb4d014f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c819c270 con 0x7fb4c807be30 2023-07-20T02:44:08.588 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.037+0000 7fb4cdeeb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4c807be30 0x7fb4c819bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:08.588 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.038+0000 7fb4cdeeb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4c807be30 0x7fb4c819bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54666/0 (socket says 172.21.15.7:54666) 2023-07-20T02:44:08.588 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.038+0000 7fb4cdeeb700 1 -- 172.21.15.7:0/3395818950 learned_addr learned my addr 172.21.15.7:0/3395818950 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:08.589 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.038+0000 7fb4cdeeb700 1 -- 172.21.15.7:0/3395818950 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c4009550 con 0x7fb4c807be30 2023-07-20T02:44:08.589 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.038+0000 7fb4cdeeb700 1 --2- 172.21.15.7:0/3395818950 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4c807be30 0x7fb4c819bd30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4006af0 tx=0x7fb4c4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:08.589 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.038+0000 7fb4beffd700 1 -- 172.21.15.7:0/3395818950 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4c4003a40 con 0x7fb4c807be30 2023-07-20T02:44:08.589 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.038+0000 7fb4beffd700 1 -- 172.21.15.7:0/3395818950 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb4c4003ba0 con 0x7fb4c807be30 2023-07-20T02:44:08.589 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.038+0000 7fb4beffd700 1 -- 172.21.15.7:0/3395818950 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4c401a6e0 con 0x7fb4c807be30 2023-07-20T02:44:08.589 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.038+0000 7fb4d014f700 1 -- 172.21.15.7:0/3395818950 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4c819c470 con 0x7fb4c807be30 2023-07-20T02:44:08.589 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.039+0000 7fb4d014f700 1 -- 172.21.15.7:0/3395818950 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4c819c890 con 0x7fb4c807be30 2023-07-20T02:44:08.590 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.039+0000 7fb4beffd700 1 -- 172.21.15.7:0/3395818950 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb4c4024430 con 0x7fb4c807be30 2023-07-20T02:44:08.590 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.039+0000 7fb4beffd700 1 -- 172.21.15.7:0/3395818950 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb4c40193c0 con 0x7fb4c807be30 2023-07-20T02:44:08.590 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.040+0000 7fb4d014f700 1 -- 172.21.15.7:0/3395818950 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4c8061da0 con 0x7fb4c807be30 2023-07-20T02:44:08.590 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.042+0000 7fb4beffd700 1 -- 172.21.15.7:0/3395818950 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fb4c401ab40 con 0x7fb4c807be30 2023-07-20T02:44:08.590 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.119+0000 7fb4d014f700 1 -- 172.21.15.7:0/3395818950 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb4c819f440 con 0x7fb4c807be30 2023-07-20T02:44:08.590 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.120+0000 7fb4beffd700 1 -- 172.21.15.7:0/3395818950 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fb4c401ace0 con 0x7fb4c807be30 2023-07-20T02:44:08.590 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.121+0000 7fb4d014f700 1 -- 172.21.15.7:0/3395818950 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4c807be30 msgr2=0x7fb4c819bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:08.590 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.121+0000 7fb4d014f700 1 --2- 172.21.15.7:0/3395818950 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4c807be30 0x7fb4c819bd30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4006af0 tx=0x7fb4c4011ad0 comp rx=0 tx=0).stop 2023-07-20T02:44:08.591 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.121+0000 7fb4d014f700 1 -- 172.21.15.7:0/3395818950 shutdown_connections 2023-07-20T02:44:08.591 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.121+0000 7fb4d014f700 1 --2- 172.21.15.7:0/3395818950 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4c807be30 0x7fb4c819bd30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:08.591 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.121+0000 7fb4d014f700 1 -- 172.21.15.7:0/3395818950 >> 172.21.15.7:0/3395818950 conn(0x7fb4c8101440 msgr2=0x7fb4c8102b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:08.591 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.121+0000 7fb4d014f700 1 -- 172.21.15.7:0/3395818950 shutdown_connections 2023-07-20T02:44:08.591 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:08.121+0000 7fb4d014f700 1 -- 172.21.15.7:0/3395818950 wait complete. 2023-07-20T02:44:08.591 INFO:teuthology.orchestra.run.smithi007.stdout:mgr not available, waiting (2/15)... 2023-07-20T02:44:08.877 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:08 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:08.513+0000 7f2db8170200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T02:44:08.877 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:08 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:08.639+0000 7f2db8170200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T02:44:08.877 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:08 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:08.869+0000 7f2db8170200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T02:44:09.543 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:09 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:09.247+0000 7f2db8170200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T02:44:09.942 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:09 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:09.535+0000 7f2db8170200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T02:44:09.942 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:09 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:09.733+0000 7f2db8170200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T02:44:10.442 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:10 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:10.085+0000 7f2db8170200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T02:44:11.442 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:11 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/525949200' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-20T02:44:11.442 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:11 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:11.079+0000 7f2db8170200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T02:44:11.442 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:11 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:11.271+0000 7f2db8170200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T02:44:11.564 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2023-07-20T02:44:11.564 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2023-07-20T02:44:11.564 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsid": "1b04643c-26a7-11ee-9b34-001a4aab830c", 2023-07-20T02:44:11.565 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "health": { 2023-07-20T02:44:11.565 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-20T02:44:11.565 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-20T02:44:11.565 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-20T02:44:11.565 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:44:11.565 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-20T02:44:11.565 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-20T02:44:11.566 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 0 2023-07-20T02:44:11.566 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2023-07-20T02:44:11.567 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-20T02:44:11.567 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "a" 2023-07-20T02:44:11.568 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2023-07-20T02:44:11.568 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2023-07-20T02:44:11.568 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-20T02:44:11.568 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T02:44:11.568 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-20T02:44:11.568 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-20T02:44:11.568 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:44:11.569 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-20T02:44:11.569 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T02:44:11.569 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-20T02:44:11.569 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-20T02:44:11.569 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-20T02:44:11.576 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-20T02:44:11.576 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-20T02:44:11.576 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-20T02:44:11.576 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:44:11.576 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-20T02:44:11.576 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-20T02:44:11.577 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-20T02:44:11.577 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-20T02:44:11.577 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-20T02:44:11.577 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-20T02:44:11.577 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-20T02:44:11.577 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-20T02:44:11.577 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-20T02:44:11.578 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:44:11.578 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-20T02:44:11.578 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T02:44:11.578 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-20T02:44:11.578 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-20T02:44:11.578 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:44:11.578 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-20T02:44:11.578 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-20T02:44:11.579 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-20T02:44:11.579 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-20T02:44:11.579 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-20T02:44:11.579 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-20T02:44:11.579 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "restful" 2023-07-20T02:44:11.579 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2023-07-20T02:44:11.579 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T02:44:11.579 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:44:11.580 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-20T02:44:11.580 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T02:44:11.580 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modified": "2023-07-20T02:43:56.545172+0000", 2023-07-20T02:44:11.580 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T02:44:11.580 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:44:11.580 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-20T02:44:11.580 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2023-07-20T02:44:11.581 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.074+0000 7fc71016f700 1 Processor -- start 2023-07-20T02:44:11.581 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.074+0000 7fc71016f700 1 -- start start 2023-07-20T02:44:11.581 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.074+0000 7fc71016f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc70807be50 0x7fc70807a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:11.581 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.074+0000 7fc71016f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70807a880 con 0x7fc70807be50 2023-07-20T02:44:11.581 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.075+0000 7fc70df0b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc70807be50 0x7fc70807a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:11.581 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.075+0000 7fc70df0b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc70807be50 0x7fc70807a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54680/0 (socket says 172.21.15.7:54680) 2023-07-20T02:44:11.581 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.075+0000 7fc70df0b700 1 -- 172.21.15.7:0/3893871319 learned_addr learned my addr 172.21.15.7:0/3893871319 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:11.582 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.075+0000 7fc70df0b700 1 -- 172.21.15.7:0/3893871319 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc70807a9c0 con 0x7fc70807be50 2023-07-20T02:44:11.582 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.076+0000 7fc70df0b700 1 --2- 172.21.15.7:0/3893871319 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc70807be50 0x7fc70807a2b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc7040098a0 tx=0x7fc704009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=539db21dddbc330a server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:11.582 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.076+0000 7fc70cf09700 1 -- 172.21.15.7:0/3893871319 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc704004160 con 0x7fc70807be50 2023-07-20T02:44:11.582 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.076+0000 7fc70cf09700 1 -- 172.21.15.7:0/3893871319 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc7040042c0 con 0x7fc70807be50 2023-07-20T02:44:11.582 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.076+0000 7fc70cf09700 1 -- 172.21.15.7:0/3893871319 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc704011a00 con 0x7fc70807be50 2023-07-20T02:44:11.582 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.076+0000 7fc71016f700 1 -- 172.21.15.7:0/3893871319 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc70807be50 msgr2=0x7fc70807a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:11.582 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.076+0000 7fc71016f700 1 --2- 172.21.15.7:0/3893871319 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc70807be50 0x7fc70807a2b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc7040098a0 tx=0x7fc704009bb0 comp rx=0 tx=0).stop 2023-07-20T02:44:11.582 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.077+0000 7fc71016f700 1 -- 172.21.15.7:0/3893871319 shutdown_connections 2023-07-20T02:44:11.583 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.077+0000 7fc71016f700 1 --2- 172.21.15.7:0/3893871319 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc70807be50 0x7fc70807a2b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:11.583 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.077+0000 7fc71016f700 1 -- 172.21.15.7:0/3893871319 >> 172.21.15.7:0/3893871319 conn(0x7fc708101440 msgr2=0x7fc708103860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:11.583 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.077+0000 7fc71016f700 1 -- 172.21.15.7:0/3893871319 shutdown_connections 2023-07-20T02:44:11.583 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.077+0000 7fc71016f700 1 -- 172.21.15.7:0/3893871319 wait complete. 2023-07-20T02:44:11.583 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.078+0000 7fc71016f700 1 Processor -- start 2023-07-20T02:44:11.583 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.078+0000 7fc71016f700 1 -- start start 2023-07-20T02:44:11.583 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.078+0000 7fc71016f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc70807be50 0x7fc70819bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:11.584 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.078+0000 7fc71016f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70819c270 con 0x7fc70807be50 2023-07-20T02:44:11.584 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.079+0000 7fc70df0b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc70807be50 0x7fc70819bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:11.584 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.079+0000 7fc70df0b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc70807be50 0x7fc70819bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54688/0 (socket says 172.21.15.7:54688) 2023-07-20T02:44:11.584 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.079+0000 7fc70df0b700 1 -- 172.21.15.7:0/525949200 learned_addr learned my addr 172.21.15.7:0/525949200 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:11.584 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.079+0000 7fc70df0b700 1 -- 172.21.15.7:0/525949200 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc704009550 con 0x7fc70807be50 2023-07-20T02:44:11.584 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.079+0000 7fc70df0b700 1 --2- 172.21.15.7:0/525949200 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc70807be50 0x7fc70819bd30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc704009fb0 tx=0x7fc704011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:11.584 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.080+0000 7fc6feffd700 1 -- 172.21.15.7:0/525949200 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc704003930 con 0x7fc70807be50 2023-07-20T02:44:11.584 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.080+0000 7fc6feffd700 1 -- 172.21.15.7:0/525949200 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc704003a90 con 0x7fc70807be50 2023-07-20T02:44:11.585 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.080+0000 7fc6feffd700 1 -- 172.21.15.7:0/525949200 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc704019610 con 0x7fc70807be50 2023-07-20T02:44:11.585 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.080+0000 7fc71016f700 1 -- 172.21.15.7:0/525949200 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc70819c470 con 0x7fc70807be50 2023-07-20T02:44:11.585 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.080+0000 7fc71016f700 1 -- 172.21.15.7:0/525949200 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc70819c890 con 0x7fc70807be50 2023-07-20T02:44:11.585 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.080+0000 7fc6feffd700 1 -- 172.21.15.7:0/525949200 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc704019b80 con 0x7fc70807be50 2023-07-20T02:44:11.585 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.081+0000 7fc6feffd700 1 -- 172.21.15.7:0/525949200 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc704019e30 con 0x7fc70807be50 2023-07-20T02:44:11.585 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.081+0000 7fc71016f700 1 -- 172.21.15.7:0/525949200 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc708061da0 con 0x7fc70807be50 2023-07-20T02:44:11.585 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.085+0000 7fc6feffd700 1 -- 172.21.15.7:0/525949200 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fc704003c00 con 0x7fc70807be50 2023-07-20T02:44:11.586 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.163+0000 7fc71016f700 1 -- 172.21.15.7:0/525949200 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fc70819f3c0 con 0x7fc70807be50 2023-07-20T02:44:11.586 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.163+0000 7fc6feffd700 1 -- 172.21.15.7:0/525949200 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fc70401aa80 con 0x7fc70807be50 2023-07-20T02:44:11.586 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.164+0000 7fc71016f700 1 -- 172.21.15.7:0/525949200 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc70807be50 msgr2=0x7fc70819bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:11.586 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.164+0000 7fc71016f700 1 --2- 172.21.15.7:0/525949200 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc70807be50 0x7fc70819bd30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc704009fb0 tx=0x7fc704011ad0 comp rx=0 tx=0).stop 2023-07-20T02:44:11.586 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.164+0000 7fc71016f700 1 -- 172.21.15.7:0/525949200 shutdown_connections 2023-07-20T02:44:11.586 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.164+0000 7fc71016f700 1 --2- 172.21.15.7:0/525949200 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc70807be50 0x7fc70819bd30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:11.586 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.164+0000 7fc71016f700 1 -- 172.21.15.7:0/525949200 >> 172.21.15.7:0/525949200 conn(0x7fc708101440 msgr2=0x7fc708102b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:11.586 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.164+0000 7fc71016f700 1 -- 172.21.15.7:0/525949200 shutdown_connections 2023-07-20T02:44:11.587 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:11.164+0000 7fc71016f700 1 -- 172.21.15.7:0/525949200 wait complete. 2023-07-20T02:44:11.587 INFO:teuthology.orchestra.run.smithi007.stdout:mgr not available, waiting (3/15)... 2023-07-20T02:44:11.942 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:11 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:11.528+0000 7f2db8170200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T02:44:11.942 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:11 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:11.687+0000 7f2db8170200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T02:44:12.442 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:11 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:11.973+0000 7f2db8170200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T02:44:13.442 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:13 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:13.084+0000 7f2db8170200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T02:44:13.443 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:13 smithi007 ceph-mon[126353]: Activating manager daemon x 2023-07-20T02:44:13.443 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:13 smithi007 ceph-mon[126353]: mgrmap e2: x(active, starting, since 0.00402112s) 2023-07-20T02:44:13.443 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:13 smithi007 ceph-mon[126353]: from='mgr.14100 172.21.15.7:0/3733624046' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-20T02:44:13.443 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:13 smithi007 ceph-mon[126353]: from='mgr.14100 172.21.15.7:0/3733624046' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-20T02:44:13.443 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:13 smithi007 ceph-mon[126353]: from='mgr.14100 172.21.15.7:0/3733624046' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-20T02:44:13.443 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:13 smithi007 ceph-mon[126353]: from='mgr.14100 172.21.15.7:0/3733624046' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:44:13.443 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:13 smithi007 ceph-mon[126353]: from='mgr.14100 172.21.15.7:0/3733624046' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-20T02:44:13.444 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:13 smithi007 ceph-mon[126353]: Manager daemon x is now available 2023-07-20T02:44:13.444 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:13 smithi007 ceph-mon[126353]: from='mgr.14100 172.21.15.7:0/3733624046' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-20T02:44:13.444 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:13 smithi007 ceph-mon[126353]: from='mgr.14100 172.21.15.7:0/3733624046' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-20T02:44:13.444 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:13 smithi007 ceph-mon[126353]: from='mgr.14100 172.21.15.7:0/3733624046' entity='mgr.x' 2023-07-20T02:44:13.444 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:13 smithi007 ceph-mon[126353]: from='mgr.14100 172.21.15.7:0/3733624046' entity='mgr.x' 2023-07-20T02:44:13.444 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:13 smithi007 ceph-mon[126353]: from='mgr.14100 172.21.15.7:0/3733624046' entity='mgr.x' 2023-07-20T02:44:14.859 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2023-07-20T02:44:14.859 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2023-07-20T02:44:14.859 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsid": "1b04643c-26a7-11ee-9b34-001a4aab830c", 2023-07-20T02:44:14.860 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "health": { 2023-07-20T02:44:14.860 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-20T02:44:14.860 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-20T02:44:14.860 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-20T02:44:14.860 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:44:14.860 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-20T02:44:14.860 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-20T02:44:14.861 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 0 2023-07-20T02:44:14.861 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2023-07-20T02:44:14.861 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-20T02:44:14.861 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "a" 2023-07-20T02:44:14.861 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2023-07-20T02:44:14.861 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2023-07-20T02:44:14.861 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-20T02:44:14.861 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T02:44:14.862 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-20T02:44:14.862 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-20T02:44:14.862 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:44:14.862 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-20T02:44:14.862 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T02:44:14.862 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-20T02:44:14.862 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-20T02:44:14.862 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-20T02:44:14.868 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-20T02:44:14.869 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-20T02:44:14.869 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-20T02:44:14.869 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:44:14.869 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-20T02:44:14.869 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-20T02:44:14.869 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-20T02:44:14.869 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-20T02:44:14.870 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-20T02:44:14.870 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-20T02:44:14.870 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-20T02:44:14.870 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-20T02:44:14.870 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-20T02:44:14.870 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:44:14.870 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-20T02:44:14.871 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T02:44:14.871 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-20T02:44:14.871 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-20T02:44:14.871 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:44:14.871 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-20T02:44:14.871 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-20T02:44:14.871 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-20T02:44:14.871 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-20T02:44:14.872 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-20T02:44:14.873 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-20T02:44:14.873 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "restful" 2023-07-20T02:44:14.873 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2023-07-20T02:44:14.873 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T02:44:14.873 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:44:14.873 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-20T02:44:14.873 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T02:44:14.874 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modified": "2023-07-20T02:43:56.545172+0000", 2023-07-20T02:44:14.874 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T02:44:14.874 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2023-07-20T02:44:14.874 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-20T02:44:14.874 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2023-07-20T02:44:14.874 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.131+0000 7f5136dd2700 1 Processor -- start 2023-07-20T02:44:14.874 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.132+0000 7f5136dd2700 1 -- start start 2023-07-20T02:44:14.874 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.132+0000 7f5136dd2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f513007be50 0x7f513007a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:14.875 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.132+0000 7f5136dd2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513007a880 con 0x7f513007be50 2023-07-20T02:44:14.875 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.132+0000 7f5134b6e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f513007be50 0x7f513007a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:14.875 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.132+0000 7f5134b6e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f513007be50 0x7f513007a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56338/0 (socket says 172.21.15.7:56338) 2023-07-20T02:44:14.875 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.132+0000 7f5134b6e700 1 -- 172.21.15.7:0/93075274 learned_addr learned my addr 172.21.15.7:0/93075274 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:14.875 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.133+0000 7f5134b6e700 1 -- 172.21.15.7:0/93075274 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513007a9c0 con 0x7f513007be50 2023-07-20T02:44:14.875 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.133+0000 7f5134b6e700 1 --2- 172.21.15.7:0/93075274 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f513007be50 0x7f513007a2b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f51240098a0 tx=0x7f5124009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f2b33df1e08bebb server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:14.875 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.133+0000 7f512f7fe700 1 -- 172.21.15.7:0/93075274 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5124004160 con 0x7f513007be50 2023-07-20T02:44:14.876 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.133+0000 7f512f7fe700 1 -- 172.21.15.7:0/93075274 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f51240042c0 con 0x7f513007be50 2023-07-20T02:44:14.876 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.133+0000 7f512f7fe700 1 -- 172.21.15.7:0/93075274 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5124011a00 con 0x7f513007be50 2023-07-20T02:44:14.876 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.134+0000 7f5136dd2700 1 -- 172.21.15.7:0/93075274 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f513007be50 msgr2=0x7f513007a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:14.876 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.134+0000 7f5136dd2700 1 --2- 172.21.15.7:0/93075274 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f513007be50 0x7f513007a2b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f51240098a0 tx=0x7f5124009bb0 comp rx=0 tx=0).stop 2023-07-20T02:44:14.876 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.134+0000 7f5136dd2700 1 -- 172.21.15.7:0/93075274 shutdown_connections 2023-07-20T02:44:14.876 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.134+0000 7f5136dd2700 1 --2- 172.21.15.7:0/93075274 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f513007be50 0x7f513007a2b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:14.876 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.134+0000 7f5136dd2700 1 -- 172.21.15.7:0/93075274 >> 172.21.15.7:0/93075274 conn(0x7f5130101440 msgr2=0x7f5130103860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:14.877 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.134+0000 7f5136dd2700 1 -- 172.21.15.7:0/93075274 shutdown_connections 2023-07-20T02:44:14.877 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.134+0000 7f5136dd2700 1 -- 172.21.15.7:0/93075274 wait complete. 2023-07-20T02:44:14.877 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.135+0000 7f5136dd2700 1 Processor -- start 2023-07-20T02:44:14.877 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.135+0000 7f5136dd2700 1 -- start start 2023-07-20T02:44:14.877 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.136+0000 7f5136dd2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f513007be50 0x7f513019bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:14.877 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.136+0000 7f5136dd2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513019c270 con 0x7f513007be50 2023-07-20T02:44:14.877 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.136+0000 7f5134b6e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f513007be50 0x7f513019bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:14.877 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.136+0000 7f5134b6e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f513007be50 0x7f513019bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56346/0 (socket says 172.21.15.7:56346) 2023-07-20T02:44:14.878 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.136+0000 7f5134b6e700 1 -- 172.21.15.7:0/4158654730 learned_addr learned my addr 172.21.15.7:0/4158654730 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:14.878 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.136+0000 7f5134b6e700 1 -- 172.21.15.7:0/4158654730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5124009550 con 0x7f513007be50 2023-07-20T02:44:14.878 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.136+0000 7f5134b6e700 1 --2- 172.21.15.7:0/4158654730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f513007be50 0x7f513019bd30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5124006af0 tx=0x7f5124011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:14.878 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.137+0000 7f512dffb700 1 -- 172.21.15.7:0/4158654730 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5124003a40 con 0x7f513007be50 2023-07-20T02:44:14.878 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.137+0000 7f512dffb700 1 -- 172.21.15.7:0/4158654730 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5124003ba0 con 0x7f513007be50 2023-07-20T02:44:14.878 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.137+0000 7f512dffb700 1 -- 172.21.15.7:0/4158654730 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f512401a6e0 con 0x7f513007be50 2023-07-20T02:44:14.878 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.137+0000 7f5136dd2700 1 -- 172.21.15.7:0/4158654730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f513019c470 con 0x7f513007be50 2023-07-20T02:44:14.879 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.137+0000 7f5136dd2700 1 -- 172.21.15.7:0/4158654730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f513019c890 con 0x7f513007be50 2023-07-20T02:44:14.879 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.138+0000 7f5136dd2700 1 -- 172.21.15.7:0/4158654730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5130061da0 con 0x7f513007be50 2023-07-20T02:44:14.879 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.138+0000 7f512dffb700 1 -- 172.21.15.7:0/4158654730 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 3) v1 ==== 45366+0+0 (secure 0 0 0) 0x7f5124024a60 con 0x7f513007be50 2023-07-20T02:44:14.879 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.138+0000 7f512dffb700 1 --2- 172.21.15.7:0/4158654730 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7f51200385e0 0x7f512003aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:14.879 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.138+0000 7f512dffb700 1 -- 172.21.15.7:0/4158654730 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f512404c490 con 0x7f513007be50 2023-07-20T02:44:14.879 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.139+0000 7f512ffff700 1 --2- 172.21.15.7:0/4158654730 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7f51200385e0 0x7f512003aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:14.879 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.139+0000 7f512ffff700 1 --2- 172.21.15.7:0/4158654730 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7f51200385e0 0x7f512003aaa0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f511c006fd0 tx=0x7f511c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:14.879 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.144+0000 7f512dffb700 1 -- 172.21.15.7:0/4158654730 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5124024430 con 0x7f513007be50 2023-07-20T02:44:14.880 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.429+0000 7f5136dd2700 1 -- 172.21.15.7:0/4158654730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f513019f9a0 con 0x7f513007be50 2023-07-20T02:44:14.880 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.430+0000 7f512dffb700 1 -- 172.21.15.7:0/4158654730 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f5124018350 con 0x7f513007be50 2023-07-20T02:44:14.880 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.432+0000 7f5136dd2700 1 -- 172.21.15.7:0/4158654730 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7f51200385e0 msgr2=0x7f512003aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:14.880 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.432+0000 7f5136dd2700 1 --2- 172.21.15.7:0/4158654730 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7f51200385e0 0x7f512003aaa0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f511c006fd0 tx=0x7f511c006e40 comp rx=0 tx=0).stop 2023-07-20T02:44:14.880 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.432+0000 7f5136dd2700 1 -- 172.21.15.7:0/4158654730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f513007be50 msgr2=0x7f513019bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:14.880 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.432+0000 7f5136dd2700 1 --2- 172.21.15.7:0/4158654730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f513007be50 0x7f513019bd30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5124006af0 tx=0x7f5124011ad0 comp rx=0 tx=0).stop 2023-07-20T02:44:14.880 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.432+0000 7f5136dd2700 1 -- 172.21.15.7:0/4158654730 shutdown_connections 2023-07-20T02:44:14.881 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.432+0000 7f5136dd2700 1 --2- 172.21.15.7:0/4158654730 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7f51200385e0 0x7f512003aaa0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:14.881 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.432+0000 7f5136dd2700 1 --2- 172.21.15.7:0/4158654730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f513007be50 0x7f513019bd30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:14.881 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.432+0000 7f5136dd2700 1 -- 172.21.15.7:0/4158654730 >> 172.21.15.7:0/4158654730 conn(0x7f5130101440 msgr2=0x7f5130102b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:14.881 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.432+0000 7f5136dd2700 1 -- 172.21.15.7:0/4158654730 shutdown_connections 2023-07-20T02:44:14.881 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:14.432+0000 7f5136dd2700 1 -- 172.21.15.7:0/4158654730 wait complete. 2023-07-20T02:44:14.881 INFO:teuthology.orchestra.run.smithi007.stdout:mgr is available 2023-07-20T02:44:15.442 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:15 smithi007 ceph-mon[126353]: mgrmap e3: x(active, since 1.01033s) 2023-07-20T02:44:15.442 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:15 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/4158654730' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-20T02:44:15.944 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2023-07-20T02:44:15.944 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [global] 2023-07-20T02:44:15.944 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout fsid = 1b04643c-26a7-11ee-9b34-001a4aab830c 2023-07-20T02:44:15.945 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-07-20T02:44:15.945 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-07-20T02:44:15.945 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-07-20T02:44:15.945 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-07-20T02:44:15.946 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2023-07-20T02:44:15.946 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [mgr] 2023-07-20T02:44:15.946 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-07-20T02:44:15.946 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2023-07-20T02:44:15.946 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [osd] 2023-07-20T02:44:15.946 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-07-20T02:44:15.946 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-07-20T02:44:15.946 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-07-20T02:44:15.947 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-07-20T02:44:15.947 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-07-20T02:44:15.947 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.355+0000 7f2ea7e9c700 1 Processor -- start 2023-07-20T02:44:15.947 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.356+0000 7f2ea7e9c700 1 -- start start 2023-07-20T02:44:15.947 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.356+0000 7f2ea7e9c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ea007be50 0x7f2ea007a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:15.947 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.356+0000 7f2ea7e9c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea007a880 con 0x7f2ea007be50 2023-07-20T02:44:15.947 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.356+0000 7f2ea5c38700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ea007be50 0x7f2ea007a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:15.947 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.356+0000 7f2ea5c38700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ea007be50 0x7f2ea007a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56348/0 (socket says 172.21.15.7:56348) 2023-07-20T02:44:15.948 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.356+0000 7f2ea5c38700 1 -- 172.21.15.7:0/2756768494 learned_addr learned my addr 172.21.15.7:0/2756768494 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:15.948 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.357+0000 7f2ea5c38700 1 -- 172.21.15.7:0/2756768494 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea007a9c0 con 0x7f2ea007be50 2023-07-20T02:44:15.948 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.357+0000 7f2ea5c38700 1 --2- 172.21.15.7:0/2756768494 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ea007be50 0x7f2ea007a2b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c0098a0 tx=0x7f2e9c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=744739080ec753d4 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:15.949 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.357+0000 7f2ea4c36700 1 -- 172.21.15.7:0/2756768494 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e9c004160 con 0x7f2ea007be50 2023-07-20T02:44:15.949 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.357+0000 7f2ea4c36700 1 -- 172.21.15.7:0/2756768494 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2e9c0042c0 con 0x7f2ea007be50 2023-07-20T02:44:15.949 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.357+0000 7f2ea4c36700 1 -- 172.21.15.7:0/2756768494 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e9c011a00 con 0x7f2ea007be50 2023-07-20T02:44:15.949 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.358+0000 7f2ea7e9c700 1 -- 172.21.15.7:0/2756768494 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ea007be50 msgr2=0x7f2ea007a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:15.949 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.358+0000 7f2ea7e9c700 1 --2- 172.21.15.7:0/2756768494 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ea007be50 0x7f2ea007a2b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c0098a0 tx=0x7f2e9c009bb0 comp rx=0 tx=0).stop 2023-07-20T02:44:15.949 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.358+0000 7f2ea7e9c700 1 -- 172.21.15.7:0/2756768494 shutdown_connections 2023-07-20T02:44:15.949 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.358+0000 7f2ea7e9c700 1 --2- 172.21.15.7:0/2756768494 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ea007be50 0x7f2ea007a2b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:15.949 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.358+0000 7f2ea7e9c700 1 -- 172.21.15.7:0/2756768494 >> 172.21.15.7:0/2756768494 conn(0x7f2ea0101440 msgr2=0x7f2ea0103860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:15.950 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.358+0000 7f2ea7e9c700 1 -- 172.21.15.7:0/2756768494 shutdown_connections 2023-07-20T02:44:15.950 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.358+0000 7f2ea7e9c700 1 -- 172.21.15.7:0/2756768494 wait complete. 2023-07-20T02:44:15.950 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.359+0000 7f2ea7e9c700 1 Processor -- start 2023-07-20T02:44:15.950 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.359+0000 7f2ea7e9c700 1 -- start start 2023-07-20T02:44:15.950 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.359+0000 7f2ea7e9c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ea007be50 0x7f2ea019bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:15.950 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.360+0000 7f2ea7e9c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea019c270 con 0x7f2ea007be50 2023-07-20T02:44:15.950 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.360+0000 7f2ea5c38700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ea007be50 0x7f2ea019bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:15.951 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.360+0000 7f2ea5c38700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ea007be50 0x7f2ea019bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56352/0 (socket says 172.21.15.7:56352) 2023-07-20T02:44:15.951 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.360+0000 7f2ea5c38700 1 -- 172.21.15.7:0/1175773100 learned_addr learned my addr 172.21.15.7:0/1175773100 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:15.951 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.360+0000 7f2ea5c38700 1 -- 172.21.15.7:0/1175773100 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e9c009550 con 0x7f2ea007be50 2023-07-20T02:44:15.951 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.360+0000 7f2ea5c38700 1 --2- 172.21.15.7:0/1175773100 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ea007be50 0x7f2ea019bd30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c006af0 tx=0x7f2e9c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:15.951 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.361+0000 7f2e96ffd700 1 -- 172.21.15.7:0/1175773100 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e9c003a40 con 0x7f2ea007be50 2023-07-20T02:44:15.951 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.361+0000 7f2e96ffd700 1 -- 172.21.15.7:0/1175773100 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2e9c003ba0 con 0x7f2ea007be50 2023-07-20T02:44:15.951 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.361+0000 7f2ea7e9c700 1 -- 172.21.15.7:0/1175773100 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ea019c470 con 0x7f2ea007be50 2023-07-20T02:44:15.951 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.361+0000 7f2e96ffd700 1 -- 172.21.15.7:0/1175773100 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e9c01a6e0 con 0x7f2ea007be50 2023-07-20T02:44:15.952 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.361+0000 7f2ea7e9c700 1 -- 172.21.15.7:0/1175773100 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ea019c890 con 0x7f2ea007be50 2023-07-20T02:44:15.952 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.362+0000 7f2e96ffd700 1 -- 172.21.15.7:0/1175773100 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f2e9c024430 con 0x7f2ea007be50 2023-07-20T02:44:15.952 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.362+0000 7f2ea7e9c700 1 -- 172.21.15.7:0/1175773100 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ea0061da0 con 0x7f2ea007be50 2023-07-20T02:44:15.952 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.362+0000 7f2e96ffd700 1 --2- 172.21.15.7:0/1175773100 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7f2e8c0386c0 0x7f2e8c03ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:15.952 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.362+0000 7f2e96ffd700 1 -- 172.21.15.7:0/1175773100 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2e9c04c6b0 con 0x7f2ea007be50 2023-07-20T02:44:15.952 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.363+0000 7f2ea5437700 1 --2- 172.21.15.7:0/1175773100 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7f2e8c0386c0 0x7f2e8c03ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:15.952 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.363+0000 7f2ea5437700 1 --2- 172.21.15.7:0/1175773100 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7f2e8c0386c0 0x7f2e8c03ab80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2e90006fd0 tx=0x7f2e90006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:15.953 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.368+0000 7f2e96ffd700 1 -- 172.21.15.7:0/1175773100 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2e9c0246e0 con 0x7f2ea007be50 2023-07-20T02:44:15.953 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.575+0000 7f2ea7e9c700 1 -- 172.21.15.7:0/1175773100 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f2ea019f710 con 0x7f2ea007be50 2023-07-20T02:44:15.953 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.576+0000 7f2e96ffd700 1 -- 172.21.15.7:0/1175773100 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f2e9c018350 con 0x7f2ea007be50 2023-07-20T02:44:15.953 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.578+0000 7f2ea7e9c700 1 -- 172.21.15.7:0/1175773100 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7f2e8c0386c0 msgr2=0x7f2e8c03ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:15.953 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.578+0000 7f2ea7e9c700 1 --2- 172.21.15.7:0/1175773100 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7f2e8c0386c0 0x7f2e8c03ab80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2e90006fd0 tx=0x7f2e90006e40 comp rx=0 tx=0).stop 2023-07-20T02:44:15.953 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.578+0000 7f2ea7e9c700 1 -- 172.21.15.7:0/1175773100 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ea007be50 msgr2=0x7f2ea019bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:15.953 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.578+0000 7f2ea7e9c700 1 --2- 172.21.15.7:0/1175773100 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ea007be50 0x7f2ea019bd30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c006af0 tx=0x7f2e9c011ad0 comp rx=0 tx=0).stop 2023-07-20T02:44:15.953 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.578+0000 7f2ea7e9c700 1 -- 172.21.15.7:0/1175773100 shutdown_connections 2023-07-20T02:44:15.954 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.578+0000 7f2ea7e9c700 1 --2- 172.21.15.7:0/1175773100 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7f2e8c0386c0 0x7f2e8c03ab80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:15.954 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.578+0000 7f2ea7e9c700 1 --2- 172.21.15.7:0/1175773100 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ea007be50 0x7f2ea019bd30 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:15.954 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.578+0000 7f2ea7e9c700 1 -- 172.21.15.7:0/1175773100 >> 172.21.15.7:0/1175773100 conn(0x7f2ea0101440 msgr2=0x7f2ea0102b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:15.954 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.578+0000 7f2ea7e9c700 1 -- 172.21.15.7:0/1175773100 shutdown_connections 2023-07-20T02:44:15.954 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:15.578+0000 7f2ea7e9c700 1 -- 172.21.15.7:0/1175773100 wait complete. 2023-07-20T02:44:15.955 INFO:teuthology.orchestra.run.smithi007.stdout:Enabling cephadm module... 2023-07-20T02:44:16.275 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:16 smithi007 ceph-mon[126353]: mgrmap e4: x(active, since 2s) 2023-07-20T02:44:16.275 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:16 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/1175773100' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-20T02:44:17.376 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:17 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/499811851' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-07-20T02:44:17.377 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:17 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: ignoring --setuser ceph since I am not root 2023-07-20T02:44:17.377 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:17 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: ignoring --setgroup ceph since I am not root 2023-07-20T02:44:17.524 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.472+0000 7ff012f23700 1 Processor -- start 2023-07-20T02:44:17.524 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.472+0000 7ff012f23700 1 -- start start 2023-07-20T02:44:17.525 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.473+0000 7ff012f23700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff00c07be90 0x7ff00c07a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:17.525 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.473+0000 7ff012f23700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00c07a8c0 con 0x7ff00c07be90 2023-07-20T02:44:17.525 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.473+0000 7ff010cbf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff00c07be90 0x7ff00c07a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:17.525 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.473+0000 7ff010cbf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff00c07be90 0x7ff00c07a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56358/0 (socket says 172.21.15.7:56358) 2023-07-20T02:44:17.525 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.473+0000 7ff010cbf700 1 -- 172.21.15.7:0/3811837537 learned_addr learned my addr 172.21.15.7:0/3811837537 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:17.525 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.474+0000 7ff010cbf700 1 -- 172.21.15.7:0/3811837537 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff00c07aa00 con 0x7ff00c07be90 2023-07-20T02:44:17.525 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.474+0000 7ff010cbf700 1 --2- 172.21.15.7:0/3811837537 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff00c07be90 0x7ff00c07a2f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff0000098a0 tx=0x7ff000009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=479a37641ced7ef3 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:17.526 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.474+0000 7ff00b7fe700 1 -- 172.21.15.7:0/3811837537 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff000004160 con 0x7ff00c07be90 2023-07-20T02:44:17.526 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.474+0000 7ff00b7fe700 1 -- 172.21.15.7:0/3811837537 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff0000042c0 con 0x7ff00c07be90 2023-07-20T02:44:17.526 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.474+0000 7ff00b7fe700 1 -- 172.21.15.7:0/3811837537 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff000011a00 con 0x7ff00c07be90 2023-07-20T02:44:17.526 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.475+0000 7ff012f23700 1 -- 172.21.15.7:0/3811837537 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff00c07be90 msgr2=0x7ff00c07a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:17.526 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.475+0000 7ff012f23700 1 --2- 172.21.15.7:0/3811837537 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff00c07be90 0x7ff00c07a2f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff0000098a0 tx=0x7ff000009bb0 comp rx=0 tx=0).stop 2023-07-20T02:44:17.526 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.475+0000 7ff012f23700 1 -- 172.21.15.7:0/3811837537 shutdown_connections 2023-07-20T02:44:17.526 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.475+0000 7ff012f23700 1 --2- 172.21.15.7:0/3811837537 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff00c07be90 0x7ff00c07a2f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:17.526 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.475+0000 7ff012f23700 1 -- 172.21.15.7:0/3811837537 >> 172.21.15.7:0/3811837537 conn(0x7ff00c101440 msgr2=0x7ff00c103860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:17.527 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.475+0000 7ff012f23700 1 -- 172.21.15.7:0/3811837537 shutdown_connections 2023-07-20T02:44:17.527 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.475+0000 7ff012f23700 1 -- 172.21.15.7:0/3811837537 wait complete. 2023-07-20T02:44:17.527 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.476+0000 7ff012f23700 1 Processor -- start 2023-07-20T02:44:17.527 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.476+0000 7ff012f23700 1 -- start start 2023-07-20T02:44:17.527 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.476+0000 7ff012f23700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff00c07be90 0x7ff00c19bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:17.527 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.476+0000 7ff012f23700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00c19c2a0 con 0x7ff00c07be90 2023-07-20T02:44:17.527 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.477+0000 7ff010cbf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff00c07be90 0x7ff00c19bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:17.528 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.477+0000 7ff010cbf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff00c07be90 0x7ff00c19bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56374/0 (socket says 172.21.15.7:56374) 2023-07-20T02:44:17.528 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.477+0000 7ff010cbf700 1 -- 172.21.15.7:0/499811851 learned_addr learned my addr 172.21.15.7:0/499811851 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:17.528 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.477+0000 7ff010cbf700 1 -- 172.21.15.7:0/499811851 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff000009550 con 0x7ff00c07be90 2023-07-20T02:44:17.528 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.477+0000 7ff010cbf700 1 --2- 172.21.15.7:0/499811851 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff00c07be90 0x7ff00c19bd60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff000009fb0 tx=0x7ff000011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:17.528 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.477+0000 7ff009ffb700 1 -- 172.21.15.7:0/499811851 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff000003930 con 0x7ff00c07be90 2023-07-20T02:44:17.528 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.477+0000 7ff009ffb700 1 -- 172.21.15.7:0/499811851 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff000003a90 con 0x7ff00c07be90 2023-07-20T02:44:17.528 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.478+0000 7ff012f23700 1 -- 172.21.15.7:0/499811851 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff00c19c4a0 con 0x7ff00c07be90 2023-07-20T02:44:17.528 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.478+0000 7ff009ffb700 1 -- 172.21.15.7:0/499811851 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff0000195e0 con 0x7ff00c07be90 2023-07-20T02:44:17.529 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.478+0000 7ff012f23700 1 -- 172.21.15.7:0/499811851 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff00c19c8c0 con 0x7ff00c07be90 2023-07-20T02:44:17.529 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.479+0000 7ff009ffb700 1 -- 172.21.15.7:0/499811851 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7ff000019850 con 0x7ff00c07be90 2023-07-20T02:44:17.529 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.479+0000 7ff012f23700 1 -- 172.21.15.7:0/499811851 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff00c061da0 con 0x7ff00c07be90 2023-07-20T02:44:17.529 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.479+0000 7ff009ffb700 1 --2- 172.21.15.7:0/499811851 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7feffc0386c0 0x7feffc03ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:17.529 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.479+0000 7ff009ffb700 1 -- 172.21.15.7:0/499811851 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff00004c5d0 con 0x7ff00c07be90 2023-07-20T02:44:17.529 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.479+0000 7ff00bfff700 1 --2- 172.21.15.7:0/499811851 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7feffc0386c0 0x7feffc03ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:17.529 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.480+0000 7ff00bfff700 1 --2- 172.21.15.7:0/499811851 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7feffc0386c0 0x7feffc03ab80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7feff8006fd0 tx=0x7feff8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:17.530 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.484+0000 7ff009ffb700 1 -- 172.21.15.7:0/499811851 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff000019b00 con 0x7ff00c07be90 2023-07-20T02:44:17.530 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:16.746+0000 7ff012f23700 1 -- 172.21.15.7:0/499811851 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7ff00c19f9d0 con 0x7ff00c07be90 2023-07-20T02:44:17.530 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:17.099+0000 7ff009ffb700 1 -- 172.21.15.7:0/499811851 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7ff00002d9b0 con 0x7ff00c07be90 2023-07-20T02:44:17.530 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:17.101+0000 7ff009ffb700 1 -- 172.21.15.7:0/499811851 <== mon.0 v2:172.21.15.7:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7ff000019d90 con 0x7ff00c07be90 2023-07-20T02:44:17.530 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:17.104+0000 7ff012f23700 1 -- 172.21.15.7:0/499811851 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7feffc0386c0 msgr2=0x7feffc03ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:17.530 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:17.104+0000 7ff012f23700 1 --2- 172.21.15.7:0/499811851 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7feffc0386c0 0x7feffc03ab80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7feff8006fd0 tx=0x7feff8006e40 comp rx=0 tx=0).stop 2023-07-20T02:44:17.530 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:17.104+0000 7ff012f23700 1 -- 172.21.15.7:0/499811851 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff00c07be90 msgr2=0x7ff00c19bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:17.530 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:17.104+0000 7ff012f23700 1 --2- 172.21.15.7:0/499811851 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff00c07be90 0x7ff00c19bd60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff000009fb0 tx=0x7ff000011ad0 comp rx=0 tx=0).stop 2023-07-20T02:44:17.531 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:17.104+0000 7ff012f23700 1 -- 172.21.15.7:0/499811851 shutdown_connections 2023-07-20T02:44:17.531 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:17.104+0000 7ff012f23700 1 --2- 172.21.15.7:0/499811851 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7feffc0386c0 0x7feffc03ab80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:17.531 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:17.104+0000 7ff012f23700 1 --2- 172.21.15.7:0/499811851 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff00c07be90 0x7ff00c19bd60 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:17.531 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:17.104+0000 7ff012f23700 1 -- 172.21.15.7:0/499811851 >> 172.21.15.7:0/499811851 conn(0x7ff00c101440 msgr2=0x7ff00c102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:17.531 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:17.104+0000 7ff012f23700 1 -- 172.21.15.7:0/499811851 shutdown_connections 2023-07-20T02:44:17.531 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:17.105+0000 7ff012f23700 1 -- 172.21.15.7:0/499811851 wait complete. 2023-07-20T02:44:17.692 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:17 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:17.369+0000 7fc1447be200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T02:44:17.692 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:17 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:17.476+0000 7fc1447be200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T02:44:17.692 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:17 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:17.612+0000 7fc1447be200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T02:44:18.106 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:17 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:17.735+0000 7fc1447be200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T02:44:18.106 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:17 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:17.972+0000 7fc1447be200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T02:44:18.383 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:18 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/499811851' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-07-20T02:44:18.384 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:18 smithi007 ceph-mon[126353]: mgrmap e5: x(active, since 4s) 2023-07-20T02:44:18.690 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:18 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:18.376+0000 7fc1447be200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T02:44:18.690 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:18 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:18.514+0000 7fc1447be200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T02:44:18.718 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2023-07-20T02:44:18.719 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-07-20T02:44:18.719 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-20T02:44:18.719 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-07-20T02:44:18.719 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-07-20T02:44:18.719 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2023-07-20T02:44:18.719 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.031+0000 7fac9df95700 1 Processor -- start 2023-07-20T02:44:18.720 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.032+0000 7fac9df95700 1 -- start start 2023-07-20T02:44:18.720 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.032+0000 7fac9df95700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac9807be50 0x7fac9807a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:18.720 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.032+0000 7fac9df95700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9807a880 con 0x7fac9807be50 2023-07-20T02:44:18.720 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.032+0000 7fac977fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac9807be50 0x7fac9807a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:18.720 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.032+0000 7fac977fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac9807be50 0x7fac9807a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56404/0 (socket says 172.21.15.7:56404) 2023-07-20T02:44:18.721 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.032+0000 7fac977fe700 1 -- 172.21.15.7:0/737156604 learned_addr learned my addr 172.21.15.7:0/737156604 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:18.721 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.033+0000 7fac977fe700 1 -- 172.21.15.7:0/737156604 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac9807a9c0 con 0x7fac9807be50 2023-07-20T02:44:18.721 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.033+0000 7fac977fe700 1 --2- 172.21.15.7:0/737156604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac9807be50 0x7fac9807a2b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fac880098a0 tx=0x7fac88009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9001b72301383860 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:18.721 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.033+0000 7fac967fc700 1 -- 172.21.15.7:0/737156604 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac88004160 con 0x7fac9807be50 2023-07-20T02:44:18.721 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.033+0000 7fac967fc700 1 -- 172.21.15.7:0/737156604 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac880042c0 con 0x7fac9807be50 2023-07-20T02:44:18.721 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.033+0000 7fac967fc700 1 -- 172.21.15.7:0/737156604 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac88011a00 con 0x7fac9807be50 2023-07-20T02:44:18.721 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.034+0000 7fac9df95700 1 -- 172.21.15.7:0/737156604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac9807be50 msgr2=0x7fac9807a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:18.722 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.034+0000 7fac9df95700 1 --2- 172.21.15.7:0/737156604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac9807be50 0x7fac9807a2b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fac880098a0 tx=0x7fac88009bb0 comp rx=0 tx=0).stop 2023-07-20T02:44:18.722 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.034+0000 7fac9df95700 1 -- 172.21.15.7:0/737156604 shutdown_connections 2023-07-20T02:44:18.722 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.034+0000 7fac9df95700 1 --2- 172.21.15.7:0/737156604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac9807be50 0x7fac9807a2b0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:18.722 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.034+0000 7fac9df95700 1 -- 172.21.15.7:0/737156604 >> 172.21.15.7:0/737156604 conn(0x7fac98101440 msgr2=0x7fac98103860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:18.722 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.034+0000 7fac9df95700 1 -- 172.21.15.7:0/737156604 shutdown_connections 2023-07-20T02:44:18.723 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.034+0000 7fac9df95700 1 -- 172.21.15.7:0/737156604 wait complete. 2023-07-20T02:44:18.723 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.035+0000 7fac9df95700 1 Processor -- start 2023-07-20T02:44:18.724 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.035+0000 7fac9df95700 1 -- start start 2023-07-20T02:44:18.724 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.035+0000 7fac9df95700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac9807be50 0x7fac9819bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:18.724 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.035+0000 7fac9df95700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9819c270 con 0x7fac9807be50 2023-07-20T02:44:18.724 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.036+0000 7fac977fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac9807be50 0x7fac9819bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:18.725 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.036+0000 7fac977fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac9807be50 0x7fac9819bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56418/0 (socket says 172.21.15.7:56418) 2023-07-20T02:44:18.725 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.036+0000 7fac977fe700 1 -- 172.21.15.7:0/4008341167 learned_addr learned my addr 172.21.15.7:0/4008341167 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:18.725 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.036+0000 7fac977fe700 1 -- 172.21.15.7:0/4008341167 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac88009550 con 0x7fac9807be50 2023-07-20T02:44:18.726 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.036+0000 7fac977fe700 1 --2- 172.21.15.7:0/4008341167 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac9807be50 0x7fac9819bd30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fac88006af0 tx=0x7fac88011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:18.727 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.037+0000 7fac94ff9700 1 -- 172.21.15.7:0/4008341167 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac88003a40 con 0x7fac9807be50 2023-07-20T02:44:18.727 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.037+0000 7fac94ff9700 1 -- 172.21.15.7:0/4008341167 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac88003ba0 con 0x7fac9807be50 2023-07-20T02:44:18.727 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.037+0000 7fac9df95700 1 -- 172.21.15.7:0/4008341167 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac9819c470 con 0x7fac9807be50 2023-07-20T02:44:18.727 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.037+0000 7fac94ff9700 1 -- 172.21.15.7:0/4008341167 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac8801a6e0 con 0x7fac9807be50 2023-07-20T02:44:18.727 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.037+0000 7fac9df95700 1 -- 172.21.15.7:0/4008341167 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac9819c890 con 0x7fac9807be50 2023-07-20T02:44:18.727 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.038+0000 7fac9df95700 1 -- 172.21.15.7:0/4008341167 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac98061da0 con 0x7fac9807be50 2023-07-20T02:44:18.728 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.038+0000 7fac94ff9700 1 -- 172.21.15.7:0/4008341167 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fac8801a860 con 0x7fac9807be50 2023-07-20T02:44:18.728 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.039+0000 7fac94ff9700 1 --2- 172.21.15.7:0/4008341167 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7fac80038760 0x7fac8003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:18.728 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.039+0000 7fac94ff9700 1 -- 172.21.15.7:0/4008341167 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fac8804c870 con 0x7fac9807be50 2023-07-20T02:44:18.728 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.039+0000 7fac96ffd700 1 -- 172.21.15.7:0/4008341167 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7fac80038760 msgr2=0x7fac8003ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/1224472946 2023-07-20T02:44:18.728 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.039+0000 7fac96ffd700 1 --2- 172.21.15.7:0/4008341167 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7fac80038760 0x7fac8003ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T02:44:18.728 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.044+0000 7fac94ff9700 1 -- 172.21.15.7:0/4008341167 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fac8801ab10 con 0x7fac9807be50 2023-07-20T02:44:18.729 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.240+0000 7fac96ffd700 1 -- 172.21.15.7:0/4008341167 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7fac80038760 msgr2=0x7fac8003ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/1224472946 2023-07-20T02:44:18.729 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.240+0000 7fac96ffd700 1 --2- 172.21.15.7:0/4008341167 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7fac80038760 0x7fac8003ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-20T02:44:18.729 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.309+0000 7fac9df95700 1 -- 172.21.15.7:0/4008341167 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fac9819f9a0 con 0x7fac9807be50 2023-07-20T02:44:18.729 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.310+0000 7fac94ff9700 1 -- 172.21.15.7:0/4008341167 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7fac88018350 con 0x7fac9807be50 2023-07-20T02:44:18.729 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.312+0000 7fac9df95700 1 -- 172.21.15.7:0/4008341167 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7fac80038760 msgr2=0x7fac8003ac20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T02:44:18.729 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.312+0000 7fac9df95700 1 --2- 172.21.15.7:0/4008341167 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7fac80038760 0x7fac8003ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:18.729 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.312+0000 7fac9df95700 1 -- 172.21.15.7:0/4008341167 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac9807be50 msgr2=0x7fac9819bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:18.730 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.312+0000 7fac9df95700 1 --2- 172.21.15.7:0/4008341167 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac9807be50 0x7fac9819bd30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fac88006af0 tx=0x7fac88011ad0 comp rx=0 tx=0).stop 2023-07-20T02:44:18.730 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.313+0000 7fac9df95700 1 -- 172.21.15.7:0/4008341167 shutdown_connections 2023-07-20T02:44:18.730 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.313+0000 7fac9df95700 1 --2- 172.21.15.7:0/4008341167 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7fac80038760 0x7fac8003ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:18.730 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.313+0000 7fac9df95700 1 --2- 172.21.15.7:0/4008341167 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac9807be50 0x7fac9819bd30 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:18.730 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.313+0000 7fac9df95700 1 -- 172.21.15.7:0/4008341167 >> 172.21.15.7:0/4008341167 conn(0x7fac98101440 msgr2=0x7fac98102b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:18.730 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.313+0000 7fac9df95700 1 -- 172.21.15.7:0/4008341167 shutdown_connections 2023-07-20T02:44:18.730 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:18.313+0000 7fac9df95700 1 -- 172.21.15.7:0/4008341167 wait complete. 2023-07-20T02:44:18.731 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for the mgr to restart... 2023-07-20T02:44:18.731 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mgr epoch 5... 2023-07-20T02:44:18.942 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:18 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:18.682+0000 7fc1447be200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T02:44:18.942 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:18 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:18.817+0000 7fc1447be200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T02:44:19.415 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:19 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/4008341167' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-20T02:44:19.692 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:19 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:19.408+0000 7fc1447be200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T02:44:20.442 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:20 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:20.172+0000 7fc1447be200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T02:44:20.442 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:20 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:20.382+0000 7fc1447be200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T02:44:21.693 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:21 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:21.355+0000 7fc1447be200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T02:44:21.693 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:21 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:21.477+0000 7fc1447be200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T02:44:22.082 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:21 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:21.705+0000 7fc1447be200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T02:44:22.365 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:22 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:22.075+0000 7fc1447be200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T02:44:22.692 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:22 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:22.358+0000 7fc1447be200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T02:44:22.693 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:22 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:22.554+0000 7fc1447be200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T02:44:23.192 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:22 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:22.902+0000 7fc1447be200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T02:44:24.192 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:23 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:23.851+0000 7fc1447be200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T02:44:24.193 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:24 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:24.028+0000 7fc1447be200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T02:44:24.692 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:24 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:24.278+0000 7fc1447be200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T02:44:24.693 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:24 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:24.427+0000 7fc1447be200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T02:44:25.192 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:24 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:24.710+0000 7fc1447be200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T02:44:26.192 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:25 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:25.801+0000 7fc1447be200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T02:44:26.193 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:25 smithi007 ceph-mon[126353]: Active manager daemon x restarted 2023-07-20T02:44:26.193 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:25 smithi007 ceph-mon[126353]: Activating manager daemon x 2023-07-20T02:44:26.193 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:25 smithi007 ceph-mon[126353]: osdmap e2: 0 total, 0 up, 0 in 2023-07-20T02:44:26.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:25 smithi007 ceph-mon[126353]: mgrmap e6: x(active, starting, since 0.00409052s) 2023-07-20T02:44:26.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:25 smithi007 ceph-mon[126353]: from='mgr.14120 172.21.15.7:0/2404405010' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:44:26.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:25 smithi007 ceph-mon[126353]: from='mgr.14120 172.21.15.7:0/2404405010' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-20T02:44:26.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:25 smithi007 ceph-mon[126353]: from='mgr.14120 172.21.15.7:0/2404405010' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-20T02:44:26.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:25 smithi007 ceph-mon[126353]: from='mgr.14120 172.21.15.7:0/2404405010' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-20T02:44:26.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:25 smithi007 ceph-mon[126353]: from='mgr.14120 172.21.15.7:0/2404405010' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-20T02:44:26.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:25 smithi007 ceph-mon[126353]: Manager daemon x is now available 2023-07-20T02:44:26.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:25 smithi007 ceph-mon[126353]: from='mgr.14120 172.21.15.7:0/2404405010' entity='mgr.x' 2023-07-20T02:44:26.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:25 smithi007 ceph-mon[126353]: from='mgr.14120 172.21.15.7:0/2404405010' entity='mgr.x' 2023-07-20T02:44:26.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:25 smithi007 ceph-mon[126353]: from='mgr.14120 172.21.15.7:0/2404405010' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:44:26.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:25 smithi007 ceph-mon[126353]: from='mgr.14120 172.21.15.7:0/2404405010' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:44:27.192 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:26 smithi007 ceph-mon[126353]: Found migration_current of "None". Setting to last migration. 2023-07-20T02:44:27.193 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:26 smithi007 ceph-mon[126353]: from='mgr.14120 172.21.15.7:0/2404405010' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:44:27.193 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:26 smithi007 ceph-mon[126353]: from='mgr.14120 172.21.15.7:0/2404405010' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-20T02:44:27.193 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:26 smithi007 ceph-mon[126353]: from='mgr.14120 172.21.15.7:0/2404405010' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-20T02:44:27.193 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:26 smithi007 ceph-mon[126353]: from='mgr.14120 172.21.15.7:0/2404405010' entity='mgr.x' 2023-07-20T02:44:27.193 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:26 smithi007 ceph-mon[126353]: from='mgr.14120 172.21.15.7:0/2404405010' entity='mgr.x' 2023-07-20T02:44:27.193 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:26 smithi007 ceph-mon[126353]: mgrmap e7: x(active, since 1.00726s) 2023-07-20T02:44:27.418 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2023-07-20T02:44:27.418 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-07-20T02:44:27.419 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "initialized": true 2023-07-20T02:44:27.419 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2023-07-20T02:44:27.419 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.316+0000 7f901be5b700 1 Processor -- start 2023-07-20T02:44:27.419 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.317+0000 7f901be5b700 1 -- start start 2023-07-20T02:44:27.419 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.317+0000 7f901be5b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f901407be70 0x7f901407a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:27.419 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.317+0000 7f901be5b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901407a8a0 con 0x7f901407be70 2023-07-20T02:44:27.419 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.317+0000 7f9019bf7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f901407be70 0x7f901407a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:27.420 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.317+0000 7f9019bf7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f901407be70 0x7f901407a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56434/0 (socket says 172.21.15.7:56434) 2023-07-20T02:44:27.420 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.317+0000 7f9019bf7700 1 -- 172.21.15.7:0/2387504861 learned_addr learned my addr 172.21.15.7:0/2387504861 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:27.420 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.318+0000 7f9019bf7700 1 -- 172.21.15.7:0/2387504861 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f901407a9e0 con 0x7f901407be70 2023-07-20T02:44:27.420 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.318+0000 7f9019bf7700 1 --2- 172.21.15.7:0/2387504861 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f901407be70 0x7f901407a2d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f90100098a0 tx=0x7f9010009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a61c8bf81fe31544 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:27.420 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.318+0000 7f9018bf5700 1 -- 172.21.15.7:0/2387504861 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9010004160 con 0x7f901407be70 2023-07-20T02:44:27.420 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.318+0000 7f9018bf5700 1 -- 172.21.15.7:0/2387504861 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f90100042c0 con 0x7f901407be70 2023-07-20T02:44:27.420 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.318+0000 7f9018bf5700 1 -- 172.21.15.7:0/2387504861 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9010011a00 con 0x7f901407be70 2023-07-20T02:44:27.421 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.319+0000 7f901be5b700 1 -- 172.21.15.7:0/2387504861 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f901407be70 msgr2=0x7f901407a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:27.421 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.319+0000 7f901be5b700 1 --2- 172.21.15.7:0/2387504861 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f901407be70 0x7f901407a2d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f90100098a0 tx=0x7f9010009bb0 comp rx=0 tx=0).stop 2023-07-20T02:44:27.421 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.319+0000 7f901be5b700 1 -- 172.21.15.7:0/2387504861 shutdown_connections 2023-07-20T02:44:27.421 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.319+0000 7f901be5b700 1 --2- 172.21.15.7:0/2387504861 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f901407be70 0x7f901407a2d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:27.421 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.319+0000 7f901be5b700 1 -- 172.21.15.7:0/2387504861 >> 172.21.15.7:0/2387504861 conn(0x7f9014101440 msgr2=0x7f9014103860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:27.421 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.319+0000 7f901be5b700 1 -- 172.21.15.7:0/2387504861 shutdown_connections 2023-07-20T02:44:27.421 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.319+0000 7f901be5b700 1 -- 172.21.15.7:0/2387504861 wait complete. 2023-07-20T02:44:27.422 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.320+0000 7f901be5b700 1 Processor -- start 2023-07-20T02:44:27.422 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.320+0000 7f901be5b700 1 -- start start 2023-07-20T02:44:27.422 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.321+0000 7f901be5b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f901407be70 0x7f901419bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:27.422 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.321+0000 7f901be5b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901419c2a0 con 0x7f901407be70 2023-07-20T02:44:27.422 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.321+0000 7f9019bf7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f901407be70 0x7f901419bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:27.422 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.321+0000 7f9019bf7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f901407be70 0x7f901419bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56442/0 (socket says 172.21.15.7:56442) 2023-07-20T02:44:27.422 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.321+0000 7f9019bf7700 1 -- 172.21.15.7:0/867654664 learned_addr learned my addr 172.21.15.7:0/867654664 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:27.422 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.321+0000 7f9019bf7700 1 -- 172.21.15.7:0/867654664 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9010009550 con 0x7f901407be70 2023-07-20T02:44:27.423 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.322+0000 7f9019bf7700 1 --2- 172.21.15.7:0/867654664 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f901407be70 0x7f901419bd60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9010006af0 tx=0x7f9010011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:27.423 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.322+0000 7f900affd700 1 -- 172.21.15.7:0/867654664 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9010003a40 con 0x7f901407be70 2023-07-20T02:44:27.423 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.322+0000 7f900affd700 1 -- 172.21.15.7:0/867654664 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9010003ba0 con 0x7f901407be70 2023-07-20T02:44:27.423 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.322+0000 7f900affd700 1 -- 172.21.15.7:0/867654664 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f901001a700 con 0x7f901407be70 2023-07-20T02:44:27.423 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.322+0000 7f901be5b700 1 -- 172.21.15.7:0/867654664 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f901419c4a0 con 0x7f901407be70 2023-07-20T02:44:27.423 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.322+0000 7f901be5b700 1 -- 172.21.15.7:0/867654664 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f901419c8c0 con 0x7f901407be70 2023-07-20T02:44:27.423 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.323+0000 7f900affd700 1 -- 172.21.15.7:0/867654664 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f901001a930 con 0x7f901407be70 2023-07-20T02:44:27.424 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.324+0000 7f900affd700 1 --2- 172.21.15.7:0/867654664 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7f9000038760 0x7f900003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:27.424 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.324+0000 7f901be5b700 1 -- 172.21.15.7:0/867654664 --> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f901410c230 con 0x7f9000038760 2023-07-20T02:44:27.424 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.324+0000 7f900affd700 1 -- 172.21.15.7:0/867654664 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f901004c840 con 0x7f901407be70 2023-07-20T02:44:27.424 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.324+0000 7f90193f6700 1 -- 172.21.15.7:0/867654664 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7f9000038760 msgr2=0x7f900003ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/1224472946 2023-07-20T02:44:27.424 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.324+0000 7f90193f6700 1 --2- 172.21.15.7:0/867654664 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7f9000038760 0x7f900003ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T02:44:27.424 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.525+0000 7f90193f6700 1 -- 172.21.15.7:0/867654664 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7f9000038760 msgr2=0x7f900003ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/1224472946 2023-07-20T02:44:27.424 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.525+0000 7f90193f6700 1 --2- 172.21.15.7:0/867654664 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7f9000038760 0x7f900003ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-20T02:44:27.425 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.926+0000 7f90193f6700 1 -- 172.21.15.7:0/867654664 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7f9000038760 msgr2=0x7f900003ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/1224472946 2023-07-20T02:44:27.425 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:19.926+0000 7f90193f6700 1 --2- 172.21.15.7:0/867654664 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7f9000038760 0x7f900003ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-20T02:44:27.425 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:20.727+0000 7f90193f6700 1 -- 172.21.15.7:0/867654664 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7f9000038760 msgr2=0x7f900003ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/1224472946 2023-07-20T02:44:27.425 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:20.727+0000 7f90193f6700 1 --2- 172.21.15.7:0/867654664 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7f9000038760 0x7f900003ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-20T02:44:27.425 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:22.328+0000 7f90193f6700 1 -- 172.21.15.7:0/867654664 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7f9000038760 msgr2=0x7f900003ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/1224472946 2023-07-20T02:44:27.425 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:22.329+0000 7f90193f6700 1 --2- 172.21.15.7:0/867654664 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7f9000038760 0x7f900003ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-07-20T02:44:27.425 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:25.532+0000 7f90193f6700 1 -- 172.21.15.7:0/867654664 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7f9000038760 msgr2=0x7f900003ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/1224472946 2023-07-20T02:44:27.426 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:25.532+0000 7f90193f6700 1 --2- 172.21.15.7:0/867654664 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7f9000038760 0x7f900003ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-07-20T02:44:27.426 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:25.806+0000 7f900affd700 1 -- 172.21.15.7:0/867654664 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mgrmap(e 6) v1 ==== 45250+0+0 (secure 0 0 0) 0x7f901002d960 con 0x7f901407be70 2023-07-20T02:44:27.426 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:25.806+0000 7f900affd700 1 -- 172.21.15.7:0/867654664 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7f9000038760 msgr2=0x7f900003ac20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T02:44:27.426 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:25.806+0000 7f900affd700 1 --2- 172.21.15.7:0/867654664 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7f9000038760 0x7f900003ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:27.426 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:26.808+0000 7f900affd700 1 -- 172.21.15.7:0/867654664 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7f901004b300 con 0x7f901407be70 2023-07-20T02:44:27.426 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:26.808+0000 7f900affd700 1 --2- 172.21.15.7:0/867654664 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f900003bfb0 0x7f900003e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:27.426 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:26.808+0000 7f900affd700 1 -- 172.21.15.7:0/867654664 --> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f901410c230 con 0x7f900003bfb0 2023-07-20T02:44:27.426 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:26.809+0000 7f90193f6700 1 --2- 172.21.15.7:0/867654664 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f900003bfb0 0x7f900003e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:27.427 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:26.810+0000 7f90193f6700 1 --2- 172.21.15.7:0/867654664 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f900003bfb0 0x7f900003e3a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9004003a10 tx=0x7f90040092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:27.427 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:26.811+0000 7f900affd700 1 -- 172.21.15.7:0/867654664 <== mgr.14120 v2:172.21.15.7:6800/2926310425 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f901410c230 con 0x7f900003bfb0 2023-07-20T02:44:27.427 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:26.818+0000 7f901be5b700 1 -- 172.21.15.7:0/867654664 --> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f901410c3a0 con 0x7f900003bfb0 2023-07-20T02:44:27.427 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:26.818+0000 7f900affd700 1 -- 172.21.15.7:0/867654664 <== mgr.14120 v2:172.21.15.7:6800/2926310425 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f901410c3a0 con 0x7f900003bfb0 2023-07-20T02:44:27.427 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:26.818+0000 7f901be5b700 1 -- 172.21.15.7:0/867654664 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f900003bfb0 msgr2=0x7f900003e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:27.427 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:26.818+0000 7f901be5b700 1 --2- 172.21.15.7:0/867654664 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f900003bfb0 0x7f900003e3a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9004003a10 tx=0x7f90040092b0 comp rx=0 tx=0).stop 2023-07-20T02:44:27.427 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:26.818+0000 7f901be5b700 1 -- 172.21.15.7:0/867654664 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f901407be70 msgr2=0x7f901419bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:27.428 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:26.818+0000 7f901be5b700 1 --2- 172.21.15.7:0/867654664 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f901407be70 0x7f901419bd60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9010006af0 tx=0x7f9010011ad0 comp rx=0 tx=0).stop 2023-07-20T02:44:27.428 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:26.818+0000 7f901be5b700 1 -- 172.21.15.7:0/867654664 shutdown_connections 2023-07-20T02:44:27.428 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:26.818+0000 7f901be5b700 1 --2- 172.21.15.7:0/867654664 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f900003bfb0 0x7f900003e3a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:27.428 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:26.818+0000 7f901be5b700 1 --2- 172.21.15.7:0/867654664 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f901407be70 0x7f901419bd60 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:27.428 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:26.818+0000 7f901be5b700 1 --2- 172.21.15.7:0/867654664 >> [v2:172.21.15.7:6800/1224472946,v1:172.21.15.7:6801/1224472946] conn(0x7f9000038760 0x7f900003ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:27.428 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:26.818+0000 7f901be5b700 1 -- 172.21.15.7:0/867654664 >> 172.21.15.7:0/867654664 conn(0x7f9014101440 msgr2=0x7f9014102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:27.428 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:26.819+0000 7f901be5b700 1 -- 172.21.15.7:0/867654664 shutdown_connections 2023-07-20T02:44:27.428 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:26.819+0000 7f901be5b700 1 -- 172.21.15.7:0/867654664 wait complete. 2023-07-20T02:44:27.429 INFO:teuthology.orchestra.run.smithi007.stdout:mgr epoch 5 is available 2023-07-20T02:44:27.429 INFO:teuthology.orchestra.run.smithi007.stdout:Setting orchestrator backend to cephadm... 2023-07-20T02:44:28.192 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:27 smithi007 ceph-mon[126353]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-20T02:44:28.193 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:27 smithi007 ceph-mon[126353]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-20T02:44:28.701 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.046+0000 7f4f1b671700 1 Processor -- start 2023-07-20T02:44:28.701 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.046+0000 7f4f1b671700 1 -- start start 2023-07-20T02:44:28.701 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.046+0000 7f4f1b671700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f1407be30 0x7f4f1407a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:28.701 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.046+0000 7f4f1b671700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1407a860 con 0x7f4f1407be30 2023-07-20T02:44:28.702 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.046+0000 7f4f1940d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f1407be30 0x7f4f1407a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:28.702 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.046+0000 7f4f1940d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f1407be30 0x7f4f1407a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59384/0 (socket says 172.21.15.7:59384) 2023-07-20T02:44:28.702 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.046+0000 7f4f1940d700 1 -- 172.21.15.7:0/2673779674 learned_addr learned my addr 172.21.15.7:0/2673779674 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:28.702 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.047+0000 7f4f1940d700 1 -- 172.21.15.7:0/2673779674 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f1407a9a0 con 0x7f4f1407be30 2023-07-20T02:44:28.702 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.047+0000 7f4f1940d700 1 --2- 172.21.15.7:0/2673779674 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f1407be30 0x7f4f1407a290 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4f100098a0 tx=0x7f4f10009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8919288581045f43 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:28.703 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.047+0000 7f4f0bfff700 1 -- 172.21.15.7:0/2673779674 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f10004160 con 0x7f4f1407be30 2023-07-20T02:44:28.703 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.047+0000 7f4f0bfff700 1 -- 172.21.15.7:0/2673779674 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f100042c0 con 0x7f4f1407be30 2023-07-20T02:44:28.703 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.047+0000 7f4f0bfff700 1 -- 172.21.15.7:0/2673779674 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f10011a00 con 0x7f4f1407be30 2023-07-20T02:44:28.703 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.048+0000 7f4f1b671700 1 -- 172.21.15.7:0/2673779674 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f1407be30 msgr2=0x7f4f1407a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:28.703 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.048+0000 7f4f1b671700 1 --2- 172.21.15.7:0/2673779674 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f1407be30 0x7f4f1407a290 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4f100098a0 tx=0x7f4f10009bb0 comp rx=0 tx=0).stop 2023-07-20T02:44:28.703 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.048+0000 7f4f1b671700 1 -- 172.21.15.7:0/2673779674 shutdown_connections 2023-07-20T02:44:28.704 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.048+0000 7f4f1b671700 1 --2- 172.21.15.7:0/2673779674 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f1407be30 0x7f4f1407a290 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:28.704 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.048+0000 7f4f1b671700 1 -- 172.21.15.7:0/2673779674 >> 172.21.15.7:0/2673779674 conn(0x7f4f14101440 msgr2=0x7f4f14103860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:28.704 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.048+0000 7f4f1b671700 1 -- 172.21.15.7:0/2673779674 shutdown_connections 2023-07-20T02:44:28.704 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.048+0000 7f4f1b671700 1 -- 172.21.15.7:0/2673779674 wait complete. 2023-07-20T02:44:28.705 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.050+0000 7f4f1b671700 1 Processor -- start 2023-07-20T02:44:28.705 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.050+0000 7f4f1b671700 1 -- start start 2023-07-20T02:44:28.705 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.051+0000 7f4f1b671700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f1407be30 0x7f4f1419bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:28.705 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.051+0000 7f4f1b671700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1419c270 con 0x7f4f1407be30 2023-07-20T02:44:28.705 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.051+0000 7f4f1940d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f1407be30 0x7f4f1419bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:28.706 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.051+0000 7f4f1940d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f1407be30 0x7f4f1419bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59394/0 (socket says 172.21.15.7:59394) 2023-07-20T02:44:28.706 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.051+0000 7f4f1940d700 1 -- 172.21.15.7:0/1063536315 learned_addr learned my addr 172.21.15.7:0/1063536315 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:28.706 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.051+0000 7f4f1940d700 1 -- 172.21.15.7:0/1063536315 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f10009550 con 0x7f4f1407be30 2023-07-20T02:44:28.706 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.051+0000 7f4f1940d700 1 --2- 172.21.15.7:0/1063536315 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f1407be30 0x7f4f1419bd30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4f10006af0 tx=0x7f4f10011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:28.706 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.052+0000 7f4f0a7fc700 1 -- 172.21.15.7:0/1063536315 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f10003a40 con 0x7f4f1407be30 2023-07-20T02:44:28.707 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.052+0000 7f4f0a7fc700 1 -- 172.21.15.7:0/1063536315 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f10003ba0 con 0x7f4f1407be30 2023-07-20T02:44:28.707 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.052+0000 7f4f1b671700 1 -- 172.21.15.7:0/1063536315 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f1419c470 con 0x7f4f1407be30 2023-07-20T02:44:28.707 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.052+0000 7f4f0a7fc700 1 -- 172.21.15.7:0/1063536315 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f1001a6e0 con 0x7f4f1407be30 2023-07-20T02:44:28.707 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.052+0000 7f4f1b671700 1 -- 172.21.15.7:0/1063536315 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f1419c890 con 0x7f4f1407be30 2023-07-20T02:44:28.707 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.053+0000 7f4f1b671700 1 -- 172.21.15.7:0/1063536315 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f14061da0 con 0x7f4f1407be30 2023-07-20T02:44:28.707 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.053+0000 7f4f0a7fc700 1 -- 172.21.15.7:0/1063536315 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f4f10024a60 con 0x7f4f1407be30 2023-07-20T02:44:28.707 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.054+0000 7f4f0a7fc700 1 --2- 172.21.15.7:0/1063536315 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f4f00038760 0x7f4f0003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:28.708 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.054+0000 7f4f0a7fc700 1 -- 172.21.15.7:0/1063536315 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f4f1004c7c0 con 0x7f4f1407be30 2023-07-20T02:44:28.708 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.054+0000 7f4f18c0c700 1 --2- 172.21.15.7:0/1063536315 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f4f00038760 0x7f4f0003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:28.708 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.054+0000 7f4f18c0c700 1 --2- 172.21.15.7:0/1063536315 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f4f00038760 0x7f4f0003ac20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4f04006fd0 tx=0x7f4f04006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:28.708 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.059+0000 7f4f0a7fc700 1 -- 172.21.15.7:0/1063536315 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4f10024430 con 0x7f4f1407be30 2023-07-20T02:44:28.709 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.280+0000 7f4f1b671700 1 -- 172.21.15.7:0/1063536315 --> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f14103770 con 0x7f4f00038760 2023-07-20T02:44:28.709 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.287+0000 7f4f0a7fc700 1 -- 172.21.15.7:0/1063536315 <== mgr.14120 v2:172.21.15.7:6800/2926310425 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4f14103770 con 0x7f4f00038760 2023-07-20T02:44:28.709 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.289+0000 7f4f1b671700 1 -- 172.21.15.7:0/1063536315 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f4f00038760 msgr2=0x7f4f0003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:28.709 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.289+0000 7f4f1b671700 1 --2- 172.21.15.7:0/1063536315 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f4f00038760 0x7f4f0003ac20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4f04006fd0 tx=0x7f4f04006e40 comp rx=0 tx=0).stop 2023-07-20T02:44:28.709 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.289+0000 7f4f1b671700 1 -- 172.21.15.7:0/1063536315 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f1407be30 msgr2=0x7f4f1419bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:28.709 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.289+0000 7f4f1b671700 1 --2- 172.21.15.7:0/1063536315 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f1407be30 0x7f4f1419bd30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4f10006af0 tx=0x7f4f10011ad0 comp rx=0 tx=0).stop 2023-07-20T02:44:28.710 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.289+0000 7f4f1b671700 1 -- 172.21.15.7:0/1063536315 shutdown_connections 2023-07-20T02:44:28.710 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.289+0000 7f4f1b671700 1 --2- 172.21.15.7:0/1063536315 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f4f00038760 0x7f4f0003ac20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:28.710 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.289+0000 7f4f1b671700 1 --2- 172.21.15.7:0/1063536315 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f1407be30 0x7f4f1419bd30 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:28.710 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.289+0000 7f4f1b671700 1 -- 172.21.15.7:0/1063536315 >> 172.21.15.7:0/1063536315 conn(0x7f4f14101440 msgr2=0x7f4f14102b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:28.710 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.290+0000 7f4f1b671700 1 -- 172.21.15.7:0/1063536315 shutdown_connections 2023-07-20T02:44:28.711 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:28.290+0000 7f4f1b671700 1 -- 172.21.15.7:0/1063536315 wait complete. 2023-07-20T02:44:28.976 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:28 smithi007 ceph-mon[126353]: mgrmap e8: x(active, since 2s) 2023-07-20T02:44:28.976 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:28 smithi007 ceph-mon[126353]: [20/Jul/2023:02:44:28] ENGINE Bus STARTING 2023-07-20T02:44:28.976 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:28 smithi007 ceph-mon[126353]: from='mgr.14120 172.21.15.7:0/2404405010' entity='mgr.x' 2023-07-20T02:44:28.977 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:28 smithi007 ceph-mon[126353]: from='mgr.14120 172.21.15.7:0/2404405010' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:44:29.882 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout value unchanged 2023-07-20T02:44:29.882 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.194+0000 7f1094064700 1 Processor -- start 2023-07-20T02:44:29.882 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.194+0000 7f1094064700 1 -- start start 2023-07-20T02:44:29.882 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.195+0000 7f1094064700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c07be30 0x7f108c07a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:29.882 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.195+0000 7f1094064700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108c07a860 con 0x7f108c07be30 2023-07-20T02:44:29.883 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.195+0000 7f1091e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c07be30 0x7f108c07a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:29.883 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.195+0000 7f1091e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c07be30 0x7f108c07a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59398/0 (socket says 172.21.15.7:59398) 2023-07-20T02:44:29.883 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.195+0000 7f1091e00700 1 -- 172.21.15.7:0/2139328253 learned_addr learned my addr 172.21.15.7:0/2139328253 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:29.883 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.195+0000 7f1091e00700 1 -- 172.21.15.7:0/2139328253 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f108c07a9a0 con 0x7f108c07be30 2023-07-20T02:44:29.883 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.196+0000 7f1091e00700 1 --2- 172.21.15.7:0/2139328253 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c07be30 0x7f108c07a290 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f10880098a0 tx=0x7f1088009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=962f6277a9184449 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:29.883 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.196+0000 7f1090dfe700 1 -- 172.21.15.7:0/2139328253 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1088004160 con 0x7f108c07be30 2023-07-20T02:44:29.884 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.196+0000 7f1090dfe700 1 -- 172.21.15.7:0/2139328253 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f10880042c0 con 0x7f108c07be30 2023-07-20T02:44:29.884 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.196+0000 7f1090dfe700 1 -- 172.21.15.7:0/2139328253 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1088011a00 con 0x7f108c07be30 2023-07-20T02:44:29.884 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.196+0000 7f1094064700 1 -- 172.21.15.7:0/2139328253 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c07be30 msgr2=0x7f108c07a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:29.884 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.196+0000 7f1094064700 1 --2- 172.21.15.7:0/2139328253 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c07be30 0x7f108c07a290 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f10880098a0 tx=0x7f1088009bb0 comp rx=0 tx=0).stop 2023-07-20T02:44:29.884 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.197+0000 7f1094064700 1 -- 172.21.15.7:0/2139328253 shutdown_connections 2023-07-20T02:44:29.884 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.197+0000 7f1094064700 1 --2- 172.21.15.7:0/2139328253 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c07be30 0x7f108c07a290 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:29.884 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.197+0000 7f1094064700 1 -- 172.21.15.7:0/2139328253 >> 172.21.15.7:0/2139328253 conn(0x7f108c101440 msgr2=0x7f108c103860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:29.885 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.197+0000 7f1094064700 1 -- 172.21.15.7:0/2139328253 shutdown_connections 2023-07-20T02:44:29.885 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.197+0000 7f1094064700 1 -- 172.21.15.7:0/2139328253 wait complete. 2023-07-20T02:44:29.885 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.198+0000 7f1094064700 1 Processor -- start 2023-07-20T02:44:29.885 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.198+0000 7f1094064700 1 -- start start 2023-07-20T02:44:29.885 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.198+0000 7f1094064700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c07be30 0x7f108c19bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:29.885 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.198+0000 7f1094064700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108c19c270 con 0x7f108c07be30 2023-07-20T02:44:29.885 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.198+0000 7f1091e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c07be30 0x7f108c19bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:29.886 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.199+0000 7f1091e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c07be30 0x7f108c19bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59400/0 (socket says 172.21.15.7:59400) 2023-07-20T02:44:29.886 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.199+0000 7f1091e00700 1 -- 172.21.15.7:0/2302121371 learned_addr learned my addr 172.21.15.7:0/2302121371 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:29.886 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.199+0000 7f1091e00700 1 -- 172.21.15.7:0/2302121371 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1088009550 con 0x7f108c07be30 2023-07-20T02:44:29.886 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.199+0000 7f1091e00700 1 --2- 172.21.15.7:0/2302121371 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c07be30 0x7f108c19bd30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1088006af0 tx=0x7f1088011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:29.886 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.199+0000 7f1082ffd700 1 -- 172.21.15.7:0/2302121371 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1088003a40 con 0x7f108c07be30 2023-07-20T02:44:29.886 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.199+0000 7f1082ffd700 1 -- 172.21.15.7:0/2302121371 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1088003ba0 con 0x7f108c07be30 2023-07-20T02:44:29.886 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.199+0000 7f1082ffd700 1 -- 172.21.15.7:0/2302121371 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f108801a6e0 con 0x7f108c07be30 2023-07-20T02:44:29.887 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.199+0000 7f1094064700 1 -- 172.21.15.7:0/2302121371 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f108c19c470 con 0x7f108c07be30 2023-07-20T02:44:29.887 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.200+0000 7f1094064700 1 -- 172.21.15.7:0/2302121371 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f108c19c890 con 0x7f108c07be30 2023-07-20T02:44:29.887 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.201+0000 7f1082ffd700 1 -- 172.21.15.7:0/2302121371 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f1088024a60 con 0x7f108c07be30 2023-07-20T02:44:29.887 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.201+0000 7f1094064700 1 -- 172.21.15.7:0/2302121371 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f108c061da0 con 0x7f108c07be30 2023-07-20T02:44:29.887 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.201+0000 7f1082ffd700 1 --2- 172.21.15.7:0/2302121371 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f1078038760 0x7f107803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:29.887 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.201+0000 7f1082ffd700 1 -- 172.21.15.7:0/2302121371 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f108804c7c0 con 0x7f108c07be30 2023-07-20T02:44:29.887 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.201+0000 7f10915ff700 1 --2- 172.21.15.7:0/2302121371 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f1078038760 0x7f107803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:29.888 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.202+0000 7f10915ff700 1 --2- 172.21.15.7:0/2302121371 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f1078038760 0x7f107803ac20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f107c006fd0 tx=0x7f107c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:29.888 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.206+0000 7f1082ffd700 1 -- 172.21.15.7:0/2302121371 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1088024430 con 0x7f108c07be30 2023-07-20T02:44:29.888 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.414+0000 7f1094064700 1 -- 172.21.15.7:0/2302121371 --> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f108c103770 con 0x7f1078038760 2023-07-20T02:44:29.888 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.415+0000 7f1082ffd700 1 -- 172.21.15.7:0/2302121371 <== mgr.14120 v2:172.21.15.7:6800/2926310425 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f108c103770 con 0x7f1078038760 2023-07-20T02:44:29.888 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.418+0000 7f1094064700 1 -- 172.21.15.7:0/2302121371 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f1078038760 msgr2=0x7f107803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:29.888 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.418+0000 7f1094064700 1 --2- 172.21.15.7:0/2302121371 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f1078038760 0x7f107803ac20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f107c006fd0 tx=0x7f107c006e40 comp rx=0 tx=0).stop 2023-07-20T02:44:29.889 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.418+0000 7f1094064700 1 -- 172.21.15.7:0/2302121371 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c07be30 msgr2=0x7f108c19bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:29.889 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.418+0000 7f1094064700 1 --2- 172.21.15.7:0/2302121371 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c07be30 0x7f108c19bd30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1088006af0 tx=0x7f1088011ad0 comp rx=0 tx=0).stop 2023-07-20T02:44:29.889 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.418+0000 7f1094064700 1 -- 172.21.15.7:0/2302121371 shutdown_connections 2023-07-20T02:44:29.889 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.418+0000 7f1094064700 1 --2- 172.21.15.7:0/2302121371 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f1078038760 0x7f107803ac20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:29.889 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.418+0000 7f1094064700 1 --2- 172.21.15.7:0/2302121371 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c07be30 0x7f108c19bd30 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:29.889 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.418+0000 7f1094064700 1 -- 172.21.15.7:0/2302121371 >> 172.21.15.7:0/2302121371 conn(0x7f108c101440 msgr2=0x7f108c102b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:29.889 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.418+0000 7f1094064700 1 -- 172.21.15.7:0/2302121371 shutdown_connections 2023-07-20T02:44:29.890 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:29.418+0000 7f1094064700 1 -- 172.21.15.7:0/2302121371 wait complete. 2023-07-20T02:44:29.890 INFO:teuthology.orchestra.run.smithi007.stdout:Generating ssh key... 2023-07-20T02:44:30.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:29 smithi007 ceph-mon[126353]: [20/Jul/2023:02:44:28] ENGINE Serving on https://172.21.15.7:7150 2023-07-20T02:44:30.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:29 smithi007 ceph-mon[126353]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:44:30.121 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:29 smithi007 ceph-mon[126353]: [20/Jul/2023:02:44:28] ENGINE Serving on http://172.21.15.7:8765 2023-07-20T02:44:30.121 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:29 smithi007 ceph-mon[126353]: [20/Jul/2023:02:44:28] ENGINE Bus STARTED 2023-07-20T02:44:31.192 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:30 smithi007 ceph-mon[126353]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:44:31.193 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:30 smithi007 ceph-mon[126353]: from='mgr.14120 172.21.15.7:0/2404405010' entity='mgr.x' 2023-07-20T02:44:31.193 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:30 smithi007 ceph-mon[126353]: from='mgr.14120 172.21.15.7:0/2404405010' entity='mgr.x' 2023-07-20T02:44:31.194 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:30 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: Generating public/private rsa key pair. 2023-07-20T02:44:31.194 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:30 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: Your identification has been saved in /tmp/tmpgi6c4qg9/key. 2023-07-20T02:44:31.194 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:30 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: Your public key has been saved in /tmp/tmpgi6c4qg9/key.pub. 2023-07-20T02:44:31.194 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:30 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: The key fingerprint is: 2023-07-20T02:44:31.194 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:30 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: SHA256:9O4lE/9+ynfl2CmXE/QynomzgJy4I+WzmoRAof4cM6M ceph-1b04643c-26a7-11ee-9b34-001a4aab830c 2023-07-20T02:44:31.194 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:30 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: The key's randomart image is: 2023-07-20T02:44:31.194 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:30 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: +---[RSA 3072]----+ 2023-07-20T02:44:31.195 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:30 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: | . | 2023-07-20T02:44:31.195 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:30 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: |. . | 2023-07-20T02:44:31.195 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:30 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: |.. . | 2023-07-20T02:44:31.195 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:30 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: |o . . . | 2023-07-20T02:44:31.195 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:30 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: |.. = S o . .| 2023-07-20T02:44:31.195 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:30 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: | .+.= .o + o o.o| 2023-07-20T02:44:31.195 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:30 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: | E.o.o. + = oo O=| 2023-07-20T02:44:31.195 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:30 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: | ...+. . =o+==*| 2023-07-20T02:44:31.196 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:30 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: | ooo+ . .+B+o| 2023-07-20T02:44:31.196 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:30 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: +----[SHA256]-----+ 2023-07-20T02:44:31.261 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.385+0000 7f63180ff700 1 Processor -- start 2023-07-20T02:44:31.262 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.385+0000 7f63180ff700 1 -- start start 2023-07-20T02:44:31.262 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.386+0000 7f63180ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f631007be50 0x7f631007a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:31.262 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.386+0000 7f63180ff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631007a880 con 0x7f631007be50 2023-07-20T02:44:31.262 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.386+0000 7f6315e9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f631007be50 0x7f631007a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:31.262 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.386+0000 7f6315e9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f631007be50 0x7f631007a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59412/0 (socket says 172.21.15.7:59412) 2023-07-20T02:44:31.262 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.386+0000 7f6315e9b700 1 -- 172.21.15.7:0/855268001 learned_addr learned my addr 172.21.15.7:0/855268001 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:31.263 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.386+0000 7f6315e9b700 1 -- 172.21.15.7:0/855268001 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f631007a9c0 con 0x7f631007be50 2023-07-20T02:44:31.263 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.387+0000 7f6315e9b700 1 --2- 172.21.15.7:0/855268001 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f631007be50 0x7f631007a2b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f630c0098a0 tx=0x7f630c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54866b63b7d09435 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:31.263 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.387+0000 7f6314e99700 1 -- 172.21.15.7:0/855268001 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f630c004160 con 0x7f631007be50 2023-07-20T02:44:31.263 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.387+0000 7f6314e99700 1 -- 172.21.15.7:0/855268001 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f630c0042c0 con 0x7f631007be50 2023-07-20T02:44:31.263 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.387+0000 7f6314e99700 1 -- 172.21.15.7:0/855268001 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f630c011a00 con 0x7f631007be50 2023-07-20T02:44:31.263 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.387+0000 7f63180ff700 1 -- 172.21.15.7:0/855268001 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f631007be50 msgr2=0x7f631007a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:31.263 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.387+0000 7f63180ff700 1 --2- 172.21.15.7:0/855268001 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f631007be50 0x7f631007a2b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f630c0098a0 tx=0x7f630c009bb0 comp rx=0 tx=0).stop 2023-07-20T02:44:31.263 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.387+0000 7f63180ff700 1 -- 172.21.15.7:0/855268001 shutdown_connections 2023-07-20T02:44:31.264 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.387+0000 7f63180ff700 1 --2- 172.21.15.7:0/855268001 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f631007be50 0x7f631007a2b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:31.264 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.387+0000 7f63180ff700 1 -- 172.21.15.7:0/855268001 >> 172.21.15.7:0/855268001 conn(0x7f6310101440 msgr2=0x7f6310103860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:31.264 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.388+0000 7f63180ff700 1 -- 172.21.15.7:0/855268001 shutdown_connections 2023-07-20T02:44:31.264 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.388+0000 7f63180ff700 1 -- 172.21.15.7:0/855268001 wait complete. 2023-07-20T02:44:31.264 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.389+0000 7f63180ff700 1 Processor -- start 2023-07-20T02:44:31.264 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.389+0000 7f63180ff700 1 -- start start 2023-07-20T02:44:31.264 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.389+0000 7f63180ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f631007be50 0x7f631019bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:31.265 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.389+0000 7f63180ff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631019c270 con 0x7f631007be50 2023-07-20T02:44:31.265 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.389+0000 7f6315e9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f631007be50 0x7f631019bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:31.265 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.389+0000 7f6315e9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f631007be50 0x7f631019bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59414/0 (socket says 172.21.15.7:59414) 2023-07-20T02:44:31.265 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.389+0000 7f6315e9b700 1 -- 172.21.15.7:0/1986583589 learned_addr learned my addr 172.21.15.7:0/1986583589 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:31.265 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.390+0000 7f6315e9b700 1 -- 172.21.15.7:0/1986583589 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f630c009550 con 0x7f631007be50 2023-07-20T02:44:31.265 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.390+0000 7f6315e9b700 1 --2- 172.21.15.7:0/1986583589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f631007be50 0x7f631019bd30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f630c006af0 tx=0x7f630c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:31.265 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.390+0000 7f6306ffd700 1 -- 172.21.15.7:0/1986583589 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f630c003a40 con 0x7f631007be50 2023-07-20T02:44:31.265 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.390+0000 7f6306ffd700 1 -- 172.21.15.7:0/1986583589 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f630c003ba0 con 0x7f631007be50 2023-07-20T02:44:31.266 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.390+0000 7f63180ff700 1 -- 172.21.15.7:0/1986583589 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f631019c470 con 0x7f631007be50 2023-07-20T02:44:31.266 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.390+0000 7f6306ffd700 1 -- 172.21.15.7:0/1986583589 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f630c01a6e0 con 0x7f631007be50 2023-07-20T02:44:31.267 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.390+0000 7f63180ff700 1 -- 172.21.15.7:0/1986583589 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f631019c890 con 0x7f631007be50 2023-07-20T02:44:31.267 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.391+0000 7f6306ffd700 1 -- 172.21.15.7:0/1986583589 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f630c024a60 con 0x7f631007be50 2023-07-20T02:44:31.267 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.391+0000 7f63180ff700 1 -- 172.21.15.7:0/1986583589 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6310061da0 con 0x7f631007be50 2023-07-20T02:44:31.267 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.392+0000 7f6306ffd700 1 --2- 172.21.15.7:0/1986583589 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f62fc038710 0x7f62fc03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:31.267 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.392+0000 7f6306ffd700 1 -- 172.21.15.7:0/1986583589 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f630c04c830 con 0x7f631007be50 2023-07-20T02:44:31.267 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.392+0000 7f631569a700 1 --2- 172.21.15.7:0/1986583589 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f62fc038710 0x7f62fc03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:31.268 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.393+0000 7f631569a700 1 --2- 172.21.15.7:0/1986583589 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f62fc038710 0x7f62fc03abd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6300006fd0 tx=0x7f6300006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:31.268 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.397+0000 7f6306ffd700 1 -- 172.21.15.7:0/1986583589 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f630c024430 con 0x7f631007be50 2023-07-20T02:44:31.268 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.605+0000 7f63180ff700 1 -- 172.21.15.7:0/1986583589 --> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f6310103770 con 0x7f62fc038710 2023-07-20T02:44:31.268 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.792+0000 7f6306ffd700 1 -- 172.21.15.7:0/1986583589 <== mgr.14120 v2:172.21.15.7:6800/2926310425 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6310103770 con 0x7f62fc038710 2023-07-20T02:44:31.268 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.795+0000 7f63180ff700 1 -- 172.21.15.7:0/1986583589 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f62fc038710 msgr2=0x7f62fc03abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:31.268 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.795+0000 7f63180ff700 1 --2- 172.21.15.7:0/1986583589 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f62fc038710 0x7f62fc03abd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6300006fd0 tx=0x7f6300006e40 comp rx=0 tx=0).stop 2023-07-20T02:44:31.268 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.795+0000 7f63180ff700 1 -- 172.21.15.7:0/1986583589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f631007be50 msgr2=0x7f631019bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:31.269 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.795+0000 7f63180ff700 1 --2- 172.21.15.7:0/1986583589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f631007be50 0x7f631019bd30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f630c006af0 tx=0x7f630c011ad0 comp rx=0 tx=0).stop 2023-07-20T02:44:31.269 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.795+0000 7f63180ff700 1 -- 172.21.15.7:0/1986583589 shutdown_connections 2023-07-20T02:44:31.269 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.795+0000 7f63180ff700 1 --2- 172.21.15.7:0/1986583589 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f62fc038710 0x7f62fc03abd0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:31.269 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.795+0000 7f63180ff700 1 --2- 172.21.15.7:0/1986583589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f631007be50 0x7f631019bd30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:31.269 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.795+0000 7f63180ff700 1 -- 172.21.15.7:0/1986583589 >> 172.21.15.7:0/1986583589 conn(0x7f6310101440 msgr2=0x7f6310102b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:31.269 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.795+0000 7f63180ff700 1 -- 172.21.15.7:0/1986583589 shutdown_connections 2023-07-20T02:44:31.269 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:30.795+0000 7f63180ff700 1 -- 172.21.15.7:0/1986583589 wait complete. 2023-07-20T02:44:32.166 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:31 smithi007 ceph-mon[126353]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:44:32.166 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:31 smithi007 ceph-mon[126353]: Generating ssh key... 2023-07-20T02:44:32.439 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdC1rUDSVXOGKnQ7A1kKZo/ocL/B66GlwTKoUkAIU/wvj3aM/qBkt0JgxclGnrHVK3Kwf1s/1xzKd39Zp16At6Ofc8k6U1NeCQREZQzrgZ/97dUM4wY14yKXNdOoFNMdxPGjiWzEQeU7kaZ1xAy3h+CsyQGMNp7PyW8yIcKUeAfZ5XM8vdYiivp5ccz/egR6xcNQjAXg0Is3ELq16eauR32cVINdLNu/qQ2FvbUJ1ckShx8IkeJZj82pvo5amWWbC3znkU+PhlkYMleu0O1UrkTwAIyCauqDbFokUZ3zvK1xt7/9uvMuKSnoXAidhiFdL4dd2k4OQYXdQ52uoloGjajplx6Zzat/Us1NeBo5nQR45Og5yaN9ZJnw/E13BC2JkcGI90hQBlXkfzpP6+lP4+Bn9hwfQaBIDalyNc10TA3moSRR58xLybWj4m0mZaY3EyXnYsPqGevY7xMlWJwj0VuW8re1ebV7W+VQQ4VcYDgg2X0jarwpqjupKt54bgnok= ceph-1b04643c-26a7-11ee-9b34-001a4aab830c 2023-07-20T02:44:32.440 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.794+0000 7fcda7126700 1 Processor -- start 2023-07-20T02:44:32.440 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.795+0000 7fcda7126700 1 -- start start 2023-07-20T02:44:32.440 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.795+0000 7fcda7126700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcda007be30 0x7fcda007a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:32.440 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.795+0000 7fcda7126700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda007a860 con 0x7fcda007be30 2023-07-20T02:44:32.440 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.795+0000 7fcda4ec2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcda007be30 0x7fcda007a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:32.440 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.795+0000 7fcda4ec2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcda007be30 0x7fcda007a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59422/0 (socket says 172.21.15.7:59422) 2023-07-20T02:44:32.441 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.795+0000 7fcda4ec2700 1 -- 172.21.15.7:0/2080190914 learned_addr learned my addr 172.21.15.7:0/2080190914 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:32.441 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.796+0000 7fcda4ec2700 1 -- 172.21.15.7:0/2080190914 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda007a9a0 con 0x7fcda007be30 2023-07-20T02:44:32.441 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.796+0000 7fcda4ec2700 1 --2- 172.21.15.7:0/2080190914 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcda007be30 0x7fcda007a290 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcd940098a0 tx=0x7fcd94009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56bce833bd70214 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:32.441 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.796+0000 7fcd9f7fe700 1 -- 172.21.15.7:0/2080190914 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd94004160 con 0x7fcda007be30 2023-07-20T02:44:32.441 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.796+0000 7fcd9f7fe700 1 -- 172.21.15.7:0/2080190914 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd940042c0 con 0x7fcda007be30 2023-07-20T02:44:32.441 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.796+0000 7fcd9f7fe700 1 -- 172.21.15.7:0/2080190914 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd94011a00 con 0x7fcda007be30 2023-07-20T02:44:32.442 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.797+0000 7fcda7126700 1 -- 172.21.15.7:0/2080190914 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcda007be30 msgr2=0x7fcda007a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:32.442 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.797+0000 7fcda7126700 1 --2- 172.21.15.7:0/2080190914 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcda007be30 0x7fcda007a290 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcd940098a0 tx=0x7fcd94009bb0 comp rx=0 tx=0).stop 2023-07-20T02:44:32.442 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.797+0000 7fcda7126700 1 -- 172.21.15.7:0/2080190914 shutdown_connections 2023-07-20T02:44:32.442 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.797+0000 7fcda7126700 1 --2- 172.21.15.7:0/2080190914 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcda007be30 0x7fcda007a290 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:32.443 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.797+0000 7fcda7126700 1 -- 172.21.15.7:0/2080190914 >> 172.21.15.7:0/2080190914 conn(0x7fcda0101440 msgr2=0x7fcda0103860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:32.443 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.797+0000 7fcda7126700 1 -- 172.21.15.7:0/2080190914 shutdown_connections 2023-07-20T02:44:32.443 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.797+0000 7fcda7126700 1 -- 172.21.15.7:0/2080190914 wait complete. 2023-07-20T02:44:32.444 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.798+0000 7fcda7126700 1 Processor -- start 2023-07-20T02:44:32.444 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.798+0000 7fcda7126700 1 -- start start 2023-07-20T02:44:32.444 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.798+0000 7fcda7126700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcda007be30 0x7fcda019bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:32.444 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.798+0000 7fcda7126700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda019c270 con 0x7fcda007be30 2023-07-20T02:44:32.445 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.799+0000 7fcda4ec2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcda007be30 0x7fcda019bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:32.445 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.799+0000 7fcda4ec2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcda007be30 0x7fcda019bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59424/0 (socket says 172.21.15.7:59424) 2023-07-20T02:44:32.445 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.799+0000 7fcda4ec2700 1 -- 172.21.15.7:0/2767977605 learned_addr learned my addr 172.21.15.7:0/2767977605 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:32.446 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.799+0000 7fcda4ec2700 1 -- 172.21.15.7:0/2767977605 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd94009550 con 0x7fcda007be30 2023-07-20T02:44:32.446 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.799+0000 7fcda4ec2700 1 --2- 172.21.15.7:0/2767977605 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcda007be30 0x7fcda019bd30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcd94006af0 tx=0x7fcd94011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:32.446 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.799+0000 7fcd9dffb700 1 -- 172.21.15.7:0/2767977605 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd94003a40 con 0x7fcda007be30 2023-07-20T02:44:32.447 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.799+0000 7fcd9dffb700 1 -- 172.21.15.7:0/2767977605 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd94003ba0 con 0x7fcda007be30 2023-07-20T02:44:32.447 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.799+0000 7fcda7126700 1 -- 172.21.15.7:0/2767977605 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcda019c470 con 0x7fcda007be30 2023-07-20T02:44:32.447 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.799+0000 7fcd9dffb700 1 -- 172.21.15.7:0/2767977605 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd9401a6e0 con 0x7fcda007be30 2023-07-20T02:44:32.448 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.800+0000 7fcda7126700 1 -- 172.21.15.7:0/2767977605 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcda019c890 con 0x7fcda007be30 2023-07-20T02:44:32.448 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.801+0000 7fcda7126700 1 -- 172.21.15.7:0/2767977605 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcda0061da0 con 0x7fcda007be30 2023-07-20T02:44:32.448 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.801+0000 7fcd9dffb700 1 -- 172.21.15.7:0/2767977605 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fcd94024a60 con 0x7fcda007be30 2023-07-20T02:44:32.449 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.801+0000 7fcd9dffb700 1 --2- 172.21.15.7:0/2767977605 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7fcd90038710 0x7fcd9003abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:32.449 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.801+0000 7fcd9dffb700 1 -- 172.21.15.7:0/2767977605 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fcd9404c830 con 0x7fcda007be30 2023-07-20T02:44:32.449 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.801+0000 7fcd9ffff700 1 --2- 172.21.15.7:0/2767977605 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7fcd90038710 0x7fcd9003abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:32.449 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.802+0000 7fcd9ffff700 1 --2- 172.21.15.7:0/2767977605 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7fcd90038710 0x7fcd9003abd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c006fd0 tx=0x7fcd8c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:32.450 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:31.806+0000 7fcd9dffb700 1 -- 172.21.15.7:0/2767977605 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcd94024430 con 0x7fcda007be30 2023-07-20T02:44:32.450 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.014+0000 7fcda7126700 1 -- 172.21.15.7:0/2767977605 --> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fcda0103770 con 0x7fcd90038710 2023-07-20T02:44:32.450 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.015+0000 7fcd9dffb700 1 -- 172.21.15.7:0/2767977605 <== mgr.14120 v2:172.21.15.7:6800/2926310425 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fcda0103770 con 0x7fcd90038710 2023-07-20T02:44:32.450 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.017+0000 7fcda7126700 1 -- 172.21.15.7:0/2767977605 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7fcd90038710 msgr2=0x7fcd9003abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:32.451 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.017+0000 7fcda7126700 1 --2- 172.21.15.7:0/2767977605 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7fcd90038710 0x7fcd9003abd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c006fd0 tx=0x7fcd8c006e40 comp rx=0 tx=0).stop 2023-07-20T02:44:32.451 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.017+0000 7fcda7126700 1 -- 172.21.15.7:0/2767977605 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcda007be30 msgr2=0x7fcda019bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:32.451 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.017+0000 7fcda7126700 1 --2- 172.21.15.7:0/2767977605 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcda007be30 0x7fcda019bd30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcd94006af0 tx=0x7fcd94011ad0 comp rx=0 tx=0).stop 2023-07-20T02:44:32.451 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.017+0000 7fcda7126700 1 -- 172.21.15.7:0/2767977605 shutdown_connections 2023-07-20T02:44:32.451 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.017+0000 7fcda7126700 1 --2- 172.21.15.7:0/2767977605 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7fcd90038710 0x7fcd9003abd0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:32.452 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.017+0000 7fcda7126700 1 --2- 172.21.15.7:0/2767977605 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcda007be30 0x7fcda019bd30 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:32.452 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.017+0000 7fcda7126700 1 -- 172.21.15.7:0/2767977605 >> 172.21.15.7:0/2767977605 conn(0x7fcda0101440 msgr2=0x7fcda0102b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:32.452 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.018+0000 7fcda7126700 1 -- 172.21.15.7:0/2767977605 shutdown_connections 2023-07-20T02:44:32.453 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.018+0000 7fcda7126700 1 -- 172.21.15.7:0/2767977605 wait complete. 2023-07-20T02:44:32.454 INFO:teuthology.orchestra.run.smithi007.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-07-20T02:44:32.454 INFO:teuthology.orchestra.run.smithi007.stdout:Adding key to root@localhost authorized_keys... 2023-07-20T02:44:32.454 INFO:teuthology.orchestra.run.smithi007.stdout:Adding host smithi007... 2023-07-20T02:44:33.192 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:32 smithi007 ceph-mon[126353]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:44:34.122 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:33 smithi007 ceph-mon[126353]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi007", "addr": "172.21.15.7", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:44:35.192 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:34 smithi007 ceph-mon[126353]: Deploying cephadm binary to smithi007 2023-07-20T02:44:36.931 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout Added host 'smithi007' with addr '172.21.15.7' 2023-07-20T02:44:36.931 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.947+0000 7fe5affb3700 1 Processor -- start 2023-07-20T02:44:36.931 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.948+0000 7fe5affb3700 1 -- start start 2023-07-20T02:44:36.932 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.948+0000 7fe5affb3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5a807be50 0x7fe5a807a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:36.932 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.948+0000 7fe5affb3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a807a880 con 0x7fe5a807be50 2023-07-20T02:44:36.932 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.948+0000 7fe5add4f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5a807be50 0x7fe5a807a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:36.932 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.948+0000 7fe5add4f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5a807be50 0x7fe5a807a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44712/0 (socket says 172.21.15.7:44712) 2023-07-20T02:44:36.932 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.948+0000 7fe5add4f700 1 -- 172.21.15.7:0/2640618134 learned_addr learned my addr 172.21.15.7:0/2640618134 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:36.932 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.949+0000 7fe5add4f700 1 -- 172.21.15.7:0/2640618134 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a807a9c0 con 0x7fe5a807be50 2023-07-20T02:44:36.932 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.949+0000 7fe5add4f700 1 --2- 172.21.15.7:0/2640618134 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5a807be50 0x7fe5a807a2b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe5a40098a0 tx=0x7fe5a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7a59060cec0289b server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:36.933 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.949+0000 7fe5acd4d700 1 -- 172.21.15.7:0/2640618134 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe5a4004160 con 0x7fe5a807be50 2023-07-20T02:44:36.933 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.949+0000 7fe5acd4d700 1 -- 172.21.15.7:0/2640618134 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe5a40042c0 con 0x7fe5a807be50 2023-07-20T02:44:36.933 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.949+0000 7fe5acd4d700 1 -- 172.21.15.7:0/2640618134 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe5a4011a00 con 0x7fe5a807be50 2023-07-20T02:44:36.933 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.950+0000 7fe5affb3700 1 -- 172.21.15.7:0/2640618134 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5a807be50 msgr2=0x7fe5a807a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:36.933 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.950+0000 7fe5affb3700 1 --2- 172.21.15.7:0/2640618134 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5a807be50 0x7fe5a807a2b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe5a40098a0 tx=0x7fe5a4009bb0 comp rx=0 tx=0).stop 2023-07-20T02:44:36.933 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.950+0000 7fe5affb3700 1 -- 172.21.15.7:0/2640618134 shutdown_connections 2023-07-20T02:44:36.933 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.950+0000 7fe5affb3700 1 --2- 172.21.15.7:0/2640618134 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5a807be50 0x7fe5a807a2b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:36.934 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.950+0000 7fe5affb3700 1 -- 172.21.15.7:0/2640618134 >> 172.21.15.7:0/2640618134 conn(0x7fe5a8101440 msgr2=0x7fe5a8103860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:36.934 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.950+0000 7fe5affb3700 1 -- 172.21.15.7:0/2640618134 shutdown_connections 2023-07-20T02:44:36.934 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.950+0000 7fe5affb3700 1 -- 172.21.15.7:0/2640618134 wait complete. 2023-07-20T02:44:36.934 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.951+0000 7fe5affb3700 1 Processor -- start 2023-07-20T02:44:36.934 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.951+0000 7fe5affb3700 1 -- start start 2023-07-20T02:44:36.934 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.952+0000 7fe5affb3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5a807be50 0x7fe5a819bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:36.934 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.952+0000 7fe5affb3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a819c270 con 0x7fe5a807be50 2023-07-20T02:44:36.934 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.952+0000 7fe5add4f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5a807be50 0x7fe5a819bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:36.935 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.952+0000 7fe5add4f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5a807be50 0x7fe5a819bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44718/0 (socket says 172.21.15.7:44718) 2023-07-20T02:44:36.935 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.952+0000 7fe5add4f700 1 -- 172.21.15.7:0/814034186 learned_addr learned my addr 172.21.15.7:0/814034186 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:36.935 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.952+0000 7fe5add4f700 1 -- 172.21.15.7:0/814034186 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a4009550 con 0x7fe5a807be50 2023-07-20T02:44:36.935 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.952+0000 7fe5add4f700 1 --2- 172.21.15.7:0/814034186 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5a807be50 0x7fe5a819bd30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4006af0 tx=0x7fe5a4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:36.935 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.953+0000 7fe59effd700 1 -- 172.21.15.7:0/814034186 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe5a4003a40 con 0x7fe5a807be50 2023-07-20T02:44:36.935 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.953+0000 7fe59effd700 1 -- 172.21.15.7:0/814034186 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe5a4003ba0 con 0x7fe5a807be50 2023-07-20T02:44:36.935 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.953+0000 7fe59effd700 1 -- 172.21.15.7:0/814034186 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe5a401a6e0 con 0x7fe5a807be50 2023-07-20T02:44:36.936 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.953+0000 7fe5affb3700 1 -- 172.21.15.7:0/814034186 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a819c470 con 0x7fe5a807be50 2023-07-20T02:44:36.936 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.953+0000 7fe5affb3700 1 -- 172.21.15.7:0/814034186 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a819c890 con 0x7fe5a807be50 2023-07-20T02:44:36.936 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.954+0000 7fe59effd700 1 -- 172.21.15.7:0/814034186 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fe5a4024a60 con 0x7fe5a807be50 2023-07-20T02:44:36.936 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.954+0000 7fe5affb3700 1 -- 172.21.15.7:0/814034186 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5a8061da0 con 0x7fe5a807be50 2023-07-20T02:44:36.936 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.954+0000 7fe59effd700 1 --2- 172.21.15.7:0/814034186 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7fe594038760 0x7fe59403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:36.936 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.954+0000 7fe59effd700 1 -- 172.21.15.7:0/814034186 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fe5a404c7e0 con 0x7fe5a807be50 2023-07-20T02:44:36.937 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.955+0000 7fe5ad54e700 1 --2- 172.21.15.7:0/814034186 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7fe594038760 0x7fe59403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:36.937 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.955+0000 7fe5ad54e700 1 --2- 172.21.15.7:0/814034186 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7fe594038760 0x7fe59403ac20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe598006fd0 tx=0x7fe598006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:36.937 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:32.959+0000 7fe59effd700 1 -- 172.21.15.7:0/814034186 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe5a4024430 con 0x7fe5a807be50 2023-07-20T02:44:36.937 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:33.173+0000 7fe5affb3700 1 -- 172.21.15.7:0/814034186 --> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi007", "addr": "172.21.15.7", "target": ["mon-mgr", ""]}) v1 -- 0x7fe5a8103770 con 0x7fe594038760 2023-07-20T02:44:36.937 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:36.386+0000 7fe59effd700 1 -- 172.21.15.7:0/814034186 <== mgr.14120 v2:172.21.15.7:6800/2926310425 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fe5a8103770 con 0x7fe594038760 2023-07-20T02:44:36.937 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:36.389+0000 7fe5affb3700 1 -- 172.21.15.7:0/814034186 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7fe594038760 msgr2=0x7fe59403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:36.937 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:36.389+0000 7fe5affb3700 1 --2- 172.21.15.7:0/814034186 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7fe594038760 0x7fe59403ac20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe598006fd0 tx=0x7fe598006e40 comp rx=0 tx=0).stop 2023-07-20T02:44:36.938 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:36.389+0000 7fe5affb3700 1 -- 172.21.15.7:0/814034186 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5a807be50 msgr2=0x7fe5a819bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:36.938 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:36.389+0000 7fe5affb3700 1 --2- 172.21.15.7:0/814034186 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5a807be50 0x7fe5a819bd30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4006af0 tx=0x7fe5a4011ad0 comp rx=0 tx=0).stop 2023-07-20T02:44:36.938 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:36.389+0000 7fe5affb3700 1 -- 172.21.15.7:0/814034186 shutdown_connections 2023-07-20T02:44:36.938 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:36.389+0000 7fe5affb3700 1 --2- 172.21.15.7:0/814034186 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7fe594038760 0x7fe59403ac20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:36.938 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:36.389+0000 7fe5affb3700 1 --2- 172.21.15.7:0/814034186 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5a807be50 0x7fe5a819bd30 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:36.938 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:36.389+0000 7fe5affb3700 1 -- 172.21.15.7:0/814034186 >> 172.21.15.7:0/814034186 conn(0x7fe5a8101440 msgr2=0x7fe5a8102b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:36.938 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:36.390+0000 7fe5affb3700 1 -- 172.21.15.7:0/814034186 shutdown_connections 2023-07-20T02:44:36.939 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:36.390+0000 7fe5affb3700 1 -- 172.21.15.7:0/814034186 wait complete. 2023-07-20T02:44:36.939 INFO:teuthology.orchestra.run.smithi007.stdout:Deploying unmanaged mon service... 2023-07-20T02:44:37.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:37 smithi007 ceph-mon[126353]: from='mgr.14120 172.21.15.7:0/2404405010' entity='mgr.x' 2023-07-20T02:44:37.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:37 smithi007 ceph-mon[126353]: Added host smithi007 2023-07-20T02:44:37.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:37 smithi007 ceph-mon[126353]: from='mgr.14120 172.21.15.7:0/2404405010' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:44:38.256 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-07-20T02:44:38.256 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.492+0000 7fad314b3700 1 Processor -- start 2023-07-20T02:44:38.256 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.492+0000 7fad314b3700 1 -- start start 2023-07-20T02:44:38.256 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.493+0000 7fad314b3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad2c07be90 0x7fad2c07a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:38.256 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.493+0000 7fad314b3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c07a8c0 con 0x7fad2c07be90 2023-07-20T02:44:38.257 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.493+0000 7fad2affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad2c07be90 0x7fad2c07a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:38.257 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.493+0000 7fad2affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad2c07be90 0x7fad2c07a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44732/0 (socket says 172.21.15.7:44732) 2023-07-20T02:44:38.257 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.493+0000 7fad2affd700 1 -- 172.21.15.7:0/1294311213 learned_addr learned my addr 172.21.15.7:0/1294311213 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:38.257 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.493+0000 7fad2affd700 1 -- 172.21.15.7:0/1294311213 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2c07aa00 con 0x7fad2c07be90 2023-07-20T02:44:38.257 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.494+0000 7fad2affd700 1 --2- 172.21.15.7:0/1294311213 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad2c07be90 0x7fad2c07a2f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fad1c0098a0 tx=0x7fad1c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6597dc3413cd15f server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:38.257 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.494+0000 7fad29ffb700 1 -- 172.21.15.7:0/1294311213 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fad1c004160 con 0x7fad2c07be90 2023-07-20T02:44:38.257 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.494+0000 7fad29ffb700 1 -- 172.21.15.7:0/1294311213 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fad1c0042c0 con 0x7fad2c07be90 2023-07-20T02:44:38.258 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.494+0000 7fad29ffb700 1 -- 172.21.15.7:0/1294311213 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fad1c011a00 con 0x7fad2c07be90 2023-07-20T02:44:38.258 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.494+0000 7fad314b3700 1 -- 172.21.15.7:0/1294311213 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad2c07be90 msgr2=0x7fad2c07a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:38.258 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.494+0000 7fad314b3700 1 --2- 172.21.15.7:0/1294311213 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad2c07be90 0x7fad2c07a2f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fad1c0098a0 tx=0x7fad1c009bb0 comp rx=0 tx=0).stop 2023-07-20T02:44:38.258 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.495+0000 7fad314b3700 1 -- 172.21.15.7:0/1294311213 shutdown_connections 2023-07-20T02:44:38.258 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.495+0000 7fad314b3700 1 --2- 172.21.15.7:0/1294311213 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad2c07be90 0x7fad2c07a2f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:38.258 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.495+0000 7fad314b3700 1 -- 172.21.15.7:0/1294311213 >> 172.21.15.7:0/1294311213 conn(0x7fad2c101440 msgr2=0x7fad2c103860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:38.258 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.495+0000 7fad314b3700 1 -- 172.21.15.7:0/1294311213 shutdown_connections 2023-07-20T02:44:38.258 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.495+0000 7fad314b3700 1 -- 172.21.15.7:0/1294311213 wait complete. 2023-07-20T02:44:38.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.496+0000 7fad314b3700 1 Processor -- start 2023-07-20T02:44:38.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.496+0000 7fad314b3700 1 -- start start 2023-07-20T02:44:38.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.496+0000 7fad314b3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad2c07be90 0x7fad2c19bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:38.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.496+0000 7fad314b3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c19c2a0 con 0x7fad2c07be90 2023-07-20T02:44:38.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.496+0000 7fad2affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad2c07be90 0x7fad2c19bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:38.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.496+0000 7fad2affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad2c07be90 0x7fad2c19bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44736/0 (socket says 172.21.15.7:44736) 2023-07-20T02:44:38.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.496+0000 7fad2affd700 1 -- 172.21.15.7:0/1306311381 learned_addr learned my addr 172.21.15.7:0/1306311381 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:38.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.497+0000 7fad2affd700 1 -- 172.21.15.7:0/1306311381 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad1c009550 con 0x7fad2c07be90 2023-07-20T02:44:38.260 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.497+0000 7fad2affd700 1 --2- 172.21.15.7:0/1306311381 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad2c07be90 0x7fad2c19bd60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fad1c009fb0 tx=0x7fad1c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:38.260 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.497+0000 7fad13fff700 1 -- 172.21.15.7:0/1306311381 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fad1c003930 con 0x7fad2c07be90 2023-07-20T02:44:38.260 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.497+0000 7fad13fff700 1 -- 172.21.15.7:0/1306311381 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fad1c003a90 con 0x7fad2c07be90 2023-07-20T02:44:38.260 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.497+0000 7fad13fff700 1 -- 172.21.15.7:0/1306311381 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fad1c0195e0 con 0x7fad2c07be90 2023-07-20T02:44:38.260 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.497+0000 7fad314b3700 1 -- 172.21.15.7:0/1306311381 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad2c19c4a0 con 0x7fad2c07be90 2023-07-20T02:44:38.260 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.497+0000 7fad314b3700 1 -- 172.21.15.7:0/1306311381 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad2c19c8c0 con 0x7fad2c07be90 2023-07-20T02:44:38.260 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.499+0000 7fad314b3700 1 -- 172.21.15.7:0/1306311381 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad2c061da0 con 0x7fad2c07be90 2023-07-20T02:44:38.261 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.499+0000 7fad13fff700 1 -- 172.21.15.7:0/1306311381 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fad1c01a9d0 con 0x7fad2c07be90 2023-07-20T02:44:38.261 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.499+0000 7fad13fff700 1 --2- 172.21.15.7:0/1306311381 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7fad140386c0 0x7fad1403ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:38.261 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.499+0000 7fad13fff700 1 -- 172.21.15.7:0/1306311381 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fad1c04c8f0 con 0x7fad2c07be90 2023-07-20T02:44:38.261 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.500+0000 7fad2a7fc700 1 --2- 172.21.15.7:0/1306311381 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7fad140386c0 0x7fad1403ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:38.261 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.500+0000 7fad2a7fc700 1 --2- 172.21.15.7:0/1306311381 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7fad140386c0 0x7fad1403ab80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fad20006fd0 tx=0x7fad20006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:38.261 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.504+0000 7fad13fff700 1 -- 172.21.15.7:0/1306311381 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fad1c023b40 con 0x7fad2c07be90 2023-07-20T02:44:38.262 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.725+0000 7fad314b3700 1 -- 172.21.15.7:0/1306311381 --> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fad2c103750 con 0x7fad140386c0 2023-07-20T02:44:38.262 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.730+0000 7fad13fff700 1 -- 172.21.15.7:0/1306311381 <== mgr.14120 v2:172.21.15.7:6800/2926310425 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fad2c103750 con 0x7fad140386c0 2023-07-20T02:44:38.262 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.733+0000 7fad314b3700 1 -- 172.21.15.7:0/1306311381 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7fad140386c0 msgr2=0x7fad1403ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:38.262 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.733+0000 7fad314b3700 1 --2- 172.21.15.7:0/1306311381 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7fad140386c0 0x7fad1403ab80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fad20006fd0 tx=0x7fad20006e40 comp rx=0 tx=0).stop 2023-07-20T02:44:38.262 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.733+0000 7fad314b3700 1 -- 172.21.15.7:0/1306311381 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad2c07be90 msgr2=0x7fad2c19bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:38.262 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.733+0000 7fad314b3700 1 --2- 172.21.15.7:0/1306311381 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad2c07be90 0x7fad2c19bd60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fad1c009fb0 tx=0x7fad1c011ad0 comp rx=0 tx=0).stop 2023-07-20T02:44:38.262 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.733+0000 7fad314b3700 1 -- 172.21.15.7:0/1306311381 shutdown_connections 2023-07-20T02:44:38.263 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.733+0000 7fad314b3700 1 --2- 172.21.15.7:0/1306311381 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7fad140386c0 0x7fad1403ab80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:38.263 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.733+0000 7fad314b3700 1 --2- 172.21.15.7:0/1306311381 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad2c07be90 0x7fad2c19bd60 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:38.263 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.733+0000 7fad314b3700 1 -- 172.21.15.7:0/1306311381 >> 172.21.15.7:0/1306311381 conn(0x7fad2c101440 msgr2=0x7fad2c102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:38.263 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.733+0000 7fad314b3700 1 -- 172.21.15.7:0/1306311381 shutdown_connections 2023-07-20T02:44:38.263 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:37.733+0000 7fad314b3700 1 -- 172.21.15.7:0/1306311381 wait complete. 2023-07-20T02:44:38.263 INFO:teuthology.orchestra.run.smithi007.stdout:Deploying unmanaged mgr service... 2023-07-20T02:44:38.942 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:38 smithi007 ceph-mon[126353]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:44:38.943 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:38 smithi007 ceph-mon[126353]: Saving service mon spec with placement count:5 2023-07-20T02:44:38.943 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:38 smithi007 ceph-mon[126353]: from='mgr.14120 172.21.15.7:0/2404405010' entity='mgr.x' 2023-07-20T02:44:39.792 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-07-20T02:44:39.792 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.088+0000 7f2fc3205700 1 Processor -- start 2023-07-20T02:44:39.792 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.088+0000 7f2fc3205700 1 -- start start 2023-07-20T02:44:39.792 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.088+0000 7f2fc3205700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2fbc07be70 0x7f2fbc07a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:39.792 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.088+0000 7f2fc3205700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fbc07a8a0 con 0x7f2fbc07be70 2023-07-20T02:44:39.793 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.089+0000 7f2fc0fa1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2fbc07be70 0x7f2fbc07a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:39.793 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.089+0000 7f2fc0fa1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2fbc07be70 0x7f2fbc07a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44748/0 (socket says 172.21.15.7:44748) 2023-07-20T02:44:39.793 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.089+0000 7f2fc0fa1700 1 -- 172.21.15.7:0/2795410107 learned_addr learned my addr 172.21.15.7:0/2795410107 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:39.793 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.089+0000 7f2fc0fa1700 1 -- 172.21.15.7:0/2795410107 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fbc07a9e0 con 0x7f2fbc07be70 2023-07-20T02:44:39.793 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.089+0000 7f2fc0fa1700 1 --2- 172.21.15.7:0/2795410107 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2fbc07be70 0x7f2fbc07a2d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2fb00098a0 tx=0x7f2fb0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5596fbb600744300 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:39.793 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.090+0000 7f2fbb7fe700 1 -- 172.21.15.7:0/2795410107 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2fb0004160 con 0x7f2fbc07be70 2023-07-20T02:44:39.793 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.090+0000 7f2fbb7fe700 1 -- 172.21.15.7:0/2795410107 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2fb00042c0 con 0x7f2fbc07be70 2023-07-20T02:44:39.794 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.090+0000 7f2fbb7fe700 1 -- 172.21.15.7:0/2795410107 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2fb0011a00 con 0x7f2fbc07be70 2023-07-20T02:44:39.794 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.090+0000 7f2fc3205700 1 -- 172.21.15.7:0/2795410107 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2fbc07be70 msgr2=0x7f2fbc07a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:39.794 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.090+0000 7f2fc3205700 1 --2- 172.21.15.7:0/2795410107 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2fbc07be70 0x7f2fbc07a2d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2fb00098a0 tx=0x7f2fb0009bb0 comp rx=0 tx=0).stop 2023-07-20T02:44:39.794 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.090+0000 7f2fc3205700 1 -- 172.21.15.7:0/2795410107 shutdown_connections 2023-07-20T02:44:39.794 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.090+0000 7f2fc3205700 1 --2- 172.21.15.7:0/2795410107 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2fbc07be70 0x7f2fbc07a2d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:39.794 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.090+0000 7f2fc3205700 1 -- 172.21.15.7:0/2795410107 >> 172.21.15.7:0/2795410107 conn(0x7f2fbc101440 msgr2=0x7f2fbc103860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:39.795 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.090+0000 7f2fc3205700 1 -- 172.21.15.7:0/2795410107 shutdown_connections 2023-07-20T02:44:39.796 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.090+0000 7f2fc3205700 1 -- 172.21.15.7:0/2795410107 wait complete. 2023-07-20T02:44:39.796 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.091+0000 7f2fc3205700 1 Processor -- start 2023-07-20T02:44:39.797 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.092+0000 7f2fc3205700 1 -- start start 2023-07-20T02:44:39.797 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.092+0000 7f2fc3205700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2fbc07be70 0x7f2fbc19bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:39.797 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.092+0000 7f2fc3205700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fbc19c2f0 con 0x7f2fbc07be70 2023-07-20T02:44:39.797 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.092+0000 7f2fc0fa1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2fbc07be70 0x7f2fbc19bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:39.797 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.092+0000 7f2fc0fa1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2fbc07be70 0x7f2fbc19bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44754/0 (socket says 172.21.15.7:44754) 2023-07-20T02:44:39.797 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.092+0000 7f2fc0fa1700 1 -- 172.21.15.7:0/1463950435 learned_addr learned my addr 172.21.15.7:0/1463950435 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:39.798 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.092+0000 7f2fc0fa1700 1 -- 172.21.15.7:0/1463950435 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb0009550 con 0x7f2fbc07be70 2023-07-20T02:44:39.798 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.093+0000 7f2fc0fa1700 1 --2- 172.21.15.7:0/1463950435 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2fbc07be70 0x7f2fbc19bdb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0009fb0 tx=0x7f2fb0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:39.798 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.093+0000 7f2fb9ffb700 1 -- 172.21.15.7:0/1463950435 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2fb0003930 con 0x7f2fbc07be70 2023-07-20T02:44:39.799 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.093+0000 7f2fb9ffb700 1 -- 172.21.15.7:0/1463950435 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2fb0003a90 con 0x7f2fbc07be70 2023-07-20T02:44:39.799 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.093+0000 7f2fb9ffb700 1 -- 172.21.15.7:0/1463950435 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2fb00195e0 con 0x7f2fbc07be70 2023-07-20T02:44:39.799 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.093+0000 7f2fc3205700 1 -- 172.21.15.7:0/1463950435 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fbc19c4f0 con 0x7f2fbc07be70 2023-07-20T02:44:39.799 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.093+0000 7f2fc3205700 1 -- 172.21.15.7:0/1463950435 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fbc19c910 con 0x7f2fbc07be70 2023-07-20T02:44:39.799 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.094+0000 7f2fc3205700 1 -- 172.21.15.7:0/1463950435 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fbc195e40 con 0x7f2fbc07be70 2023-07-20T02:44:39.800 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.094+0000 7f2fb9ffb700 1 -- 172.21.15.7:0/1463950435 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f2fb001a9d0 con 0x7f2fbc07be70 2023-07-20T02:44:39.800 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.095+0000 7f2fb9ffb700 1 --2- 172.21.15.7:0/1463950435 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f2fac038710 0x7f2fac03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:39.800 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.095+0000 7f2fb9ffb700 1 -- 172.21.15.7:0/1463950435 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f2fb004c7d0 con 0x7f2fbc07be70 2023-07-20T02:44:39.800 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.095+0000 7f2fbbfff700 1 --2- 172.21.15.7:0/1463950435 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f2fac038710 0x7f2fac03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:39.800 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.096+0000 7f2fbbfff700 1 --2- 172.21.15.7:0/1463950435 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f2fac038710 0x7f2fac03abd0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8006fd0 tx=0x7f2fa8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:39.800 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.100+0000 7f2fb9ffb700 1 -- 172.21.15.7:0/1463950435 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2fb0023b40 con 0x7f2fbc07be70 2023-07-20T02:44:39.801 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.167+0000 7f2fb9ffb700 1 -- 172.21.15.7:0/1463950435 <== mon.0 v2:172.21.15.7:3300/0 7 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2fb004c1e0 con 0x7f2fbc07be70 2023-07-20T02:44:39.801 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.330+0000 7f2fc3205700 1 -- 172.21.15.7:0/1463950435 --> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f2fbc060bb0 con 0x7f2fac038710 2023-07-20T02:44:39.801 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.335+0000 7f2fb9ffb700 1 -- 172.21.15.7:0/1463950435 <== mgr.14120 v2:172.21.15.7:6800/2926310425 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f2fbc060bb0 con 0x7f2fac038710 2023-07-20T02:44:39.801 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.338+0000 7f2fc3205700 1 -- 172.21.15.7:0/1463950435 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f2fac038710 msgr2=0x7f2fac03abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:39.802 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.338+0000 7f2fc3205700 1 --2- 172.21.15.7:0/1463950435 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f2fac038710 0x7f2fac03abd0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8006fd0 tx=0x7f2fa8006e40 comp rx=0 tx=0).stop 2023-07-20T02:44:39.802 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.338+0000 7f2fc3205700 1 -- 172.21.15.7:0/1463950435 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2fbc07be70 msgr2=0x7f2fbc19bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:39.802 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.338+0000 7f2fc3205700 1 --2- 172.21.15.7:0/1463950435 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2fbc07be70 0x7f2fbc19bdb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0009fb0 tx=0x7f2fb0011ad0 comp rx=0 tx=0).stop 2023-07-20T02:44:39.802 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.338+0000 7f2fc3205700 1 -- 172.21.15.7:0/1463950435 shutdown_connections 2023-07-20T02:44:39.802 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.338+0000 7f2fc3205700 1 --2- 172.21.15.7:0/1463950435 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f2fac038710 0x7f2fac03abd0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:39.802 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.338+0000 7f2fc3205700 1 --2- 172.21.15.7:0/1463950435 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2fbc07be70 0x7f2fbc19bdb0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:39.802 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.338+0000 7f2fc3205700 1 -- 172.21.15.7:0/1463950435 >> 172.21.15.7:0/1463950435 conn(0x7f2fbc101440 msgr2=0x7f2fbc102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:39.803 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.338+0000 7f2fc3205700 1 -- 172.21.15.7:0/1463950435 shutdown_connections 2023-07-20T02:44:39.803 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:39.338+0000 7f2fc3205700 1 -- 172.21.15.7:0/1463950435 wait complete. 2023-07-20T02:44:40.442 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:40 smithi007 ceph-mon[126353]: from='mgr.14120 172.21.15.7:0/2404405010' entity='mgr.x' 2023-07-20T02:44:40.443 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:40 smithi007 ceph-mon[126353]: from='mgr.14120 172.21.15.7:0/2404405010' entity='mgr.x' 2023-07-20T02:44:40.443 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:40 smithi007 ceph-mon[126353]: from='mgr.14120 172.21.15.7:0/2404405010' entity='mgr.x' 2023-07-20T02:44:41.080 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.356+0000 7f8443527700 1 Processor -- start 2023-07-20T02:44:41.081 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.356+0000 7f8443527700 1 -- start start 2023-07-20T02:44:41.081 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.356+0000 7f8443527700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f843c07be30 0x7f843c07a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:41.081 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.356+0000 7f8443527700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f843c07a860 con 0x7f843c07be30 2023-07-20T02:44:41.081 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.357+0000 7f84412c3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f843c07be30 0x7f843c07a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:41.081 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.357+0000 7f84412c3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f843c07be30 0x7f843c07a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44768/0 (socket says 172.21.15.7:44768) 2023-07-20T02:44:41.081 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.357+0000 7f84412c3700 1 -- 172.21.15.7:0/4036022273 learned_addr learned my addr 172.21.15.7:0/4036022273 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:41.081 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.357+0000 7f84412c3700 1 -- 172.21.15.7:0/4036022273 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f843c07a9a0 con 0x7f843c07be30 2023-07-20T02:44:41.082 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.357+0000 7f84412c3700 1 --2- 172.21.15.7:0/4036022273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f843c07be30 0x7f843c07a290 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f84380098a0 tx=0x7f8438009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a20a39941273353 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:41.082 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.358+0000 7f8433fff700 1 -- 172.21.15.7:0/4036022273 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8438004160 con 0x7f843c07be30 2023-07-20T02:44:41.082 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.358+0000 7f8433fff700 1 -- 172.21.15.7:0/4036022273 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f84380042c0 con 0x7f843c07be30 2023-07-20T02:44:41.082 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.358+0000 7f8433fff700 1 -- 172.21.15.7:0/4036022273 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f84380119b0 con 0x7f843c07be30 2023-07-20T02:44:41.082 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.358+0000 7f8443527700 1 -- 172.21.15.7:0/4036022273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f843c07be30 msgr2=0x7f843c07a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:41.082 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.358+0000 7f8443527700 1 --2- 172.21.15.7:0/4036022273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f843c07be30 0x7f843c07a290 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f84380098a0 tx=0x7f8438009bb0 comp rx=0 tx=0).stop 2023-07-20T02:44:41.082 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.358+0000 7f8443527700 1 -- 172.21.15.7:0/4036022273 shutdown_connections 2023-07-20T02:44:41.083 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.358+0000 7f8443527700 1 --2- 172.21.15.7:0/4036022273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f843c07be30 0x7f843c07a290 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:41.083 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.358+0000 7f8443527700 1 -- 172.21.15.7:0/4036022273 >> 172.21.15.7:0/4036022273 conn(0x7f843c101440 msgr2=0x7f843c103860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:41.083 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.359+0000 7f8443527700 1 -- 172.21.15.7:0/4036022273 shutdown_connections 2023-07-20T02:44:41.083 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.359+0000 7f8443527700 1 -- 172.21.15.7:0/4036022273 wait complete. 2023-07-20T02:44:41.083 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.360+0000 7f8443527700 1 Processor -- start 2023-07-20T02:44:41.083 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.360+0000 7f8443527700 1 -- start start 2023-07-20T02:44:41.083 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.360+0000 7f8443527700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f843c07be30 0x7f843c19bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:41.086 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.360+0000 7f8443527700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f843c19c2f0 con 0x7f843c07be30 2023-07-20T02:44:41.086 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.360+0000 7f84412c3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f843c07be30 0x7f843c19bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:41.086 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.360+0000 7f84412c3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f843c07be30 0x7f843c19bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44770/0 (socket says 172.21.15.7:44770) 2023-07-20T02:44:41.086 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.360+0000 7f84412c3700 1 -- 172.21.15.7:0/1904410412 learned_addr learned my addr 172.21.15.7:0/1904410412 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:41.086 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.361+0000 7f84412c3700 1 -- 172.21.15.7:0/1904410412 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8438009550 con 0x7f843c07be30 2023-07-20T02:44:41.086 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.361+0000 7f84412c3700 1 --2- 172.21.15.7:0/1904410412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f843c07be30 0x7f843c19bdb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8438011980 tx=0x7f8438011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:41.087 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.361+0000 7f84327fc700 1 -- 172.21.15.7:0/1904410412 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8438003a60 con 0x7f843c07be30 2023-07-20T02:44:41.087 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.361+0000 7f8443527700 1 -- 172.21.15.7:0/1904410412 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f843c19c4f0 con 0x7f843c07be30 2023-07-20T02:44:41.087 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.361+0000 7f84327fc700 1 -- 172.21.15.7:0/1904410412 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8438003bc0 con 0x7f843c07be30 2023-07-20T02:44:41.087 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.361+0000 7f84327fc700 1 -- 172.21.15.7:0/1904410412 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f843801a670 con 0x7f843c07be30 2023-07-20T02:44:41.087 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.361+0000 7f8443527700 1 -- 172.21.15.7:0/1904410412 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f843c19c910 con 0x7f843c07be30 2023-07-20T02:44:41.087 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.362+0000 7f8443527700 1 -- 172.21.15.7:0/1904410412 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f843c061da0 con 0x7f843c07be30 2023-07-20T02:44:41.087 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.362+0000 7f84327fc700 1 -- 172.21.15.7:0/1904410412 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f8438024a00 con 0x7f843c07be30 2023-07-20T02:44:41.088 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.363+0000 7f84327fc700 1 --2- 172.21.15.7:0/1904410412 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f8428038760 0x7f842803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:41.088 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.363+0000 7f84327fc700 1 -- 172.21.15.7:0/1904410412 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f843804c8a0 con 0x7f843c07be30 2023-07-20T02:44:41.088 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.363+0000 7f8440ac2700 1 --2- 172.21.15.7:0/1904410412 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f8428038760 0x7f842803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:41.088 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.364+0000 7f8440ac2700 1 --2- 172.21.15.7:0/1904410412 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f8428038760 0x7f842803ac20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f842c006fd0 tx=0x7f842c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:41.088 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.368+0000 7f84327fc700 1 -- 172.21.15.7:0/1904410412 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8438024440 con 0x7f843c07be30 2023-07-20T02:44:41.088 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.583+0000 7f8443527700 1 -- 172.21.15.7:0/1904410412 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f843c19f350 con 0x7f843c07be30 2023-07-20T02:44:41.088 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.587+0000 7f84327fc700 1 -- 172.21.15.7:0/1904410412 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f8438018350 con 0x7f843c07be30 2023-07-20T02:44:41.088 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.590+0000 7f8443527700 1 -- 172.21.15.7:0/1904410412 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f8428038760 msgr2=0x7f842803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:41.089 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.590+0000 7f8443527700 1 --2- 172.21.15.7:0/1904410412 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f8428038760 0x7f842803ac20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f842c006fd0 tx=0x7f842c006e40 comp rx=0 tx=0).stop 2023-07-20T02:44:41.089 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.590+0000 7f8443527700 1 -- 172.21.15.7:0/1904410412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f843c07be30 msgr2=0x7f843c19bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:41.089 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.590+0000 7f8443527700 1 --2- 172.21.15.7:0/1904410412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f843c07be30 0x7f843c19bdb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8438011980 tx=0x7f8438011a60 comp rx=0 tx=0).stop 2023-07-20T02:44:41.089 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.591+0000 7f8443527700 1 -- 172.21.15.7:0/1904410412 shutdown_connections 2023-07-20T02:44:41.089 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.591+0000 7f8443527700 1 --2- 172.21.15.7:0/1904410412 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f8428038760 0x7f842803ac20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:41.089 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.591+0000 7f8443527700 1 --2- 172.21.15.7:0/1904410412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f843c07be30 0x7f843c19bdb0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:41.089 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.591+0000 7f8443527700 1 -- 172.21.15.7:0/1904410412 >> 172.21.15.7:0/1904410412 conn(0x7f843c101440 msgr2=0x7f843c102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:41.090 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.591+0000 7f8443527700 1 -- 172.21.15.7:0/1904410412 shutdown_connections 2023-07-20T02:44:41.090 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:40.591+0000 7f8443527700 1 -- 172.21.15.7:0/1904410412 wait complete. 2023-07-20T02:44:41.443 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:41 smithi007 ceph-mon[126353]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:44:41.443 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:41 smithi007 ceph-mon[126353]: Saving service mgr spec with placement count:2 2023-07-20T02:44:41.444 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:41 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/1904410412' entity='client.admin' 2023-07-20T02:44:42.668 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.058+0000 7fe3941d1700 1 Processor -- start 2023-07-20T02:44:42.669 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.059+0000 7fe3941d1700 1 -- start start 2023-07-20T02:44:42.669 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.059+0000 7fe3941d1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe38c106940 0x7fe38c108d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:42.670 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.059+0000 7fe3941d1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38c079e00 con 0x7fe38c106940 2023-07-20T02:44:42.670 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.059+0000 7fe391f6d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe38c106940 0x7fe38c108d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:42.671 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.059+0000 7fe391f6d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe38c106940 0x7fe38c108d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49660/0 (socket says 172.21.15.7:49660) 2023-07-20T02:44:42.671 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.059+0000 7fe391f6d700 1 -- 172.21.15.7:0/2870145156 learned_addr learned my addr 172.21.15.7:0/2870145156 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:42.671 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.060+0000 7fe391f6d700 1 -- 172.21.15.7:0/2870145156 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe38c109270 con 0x7fe38c106940 2023-07-20T02:44:42.671 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.060+0000 7fe391f6d700 1 --2- 172.21.15.7:0/2870145156 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe38c106940 0x7fe38c108d30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe3880098a0 tx=0x7fe388009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ddaeb00878b8cb2 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:42.672 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.060+0000 7fe390f6b700 1 -- 172.21.15.7:0/2870145156 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe388004160 con 0x7fe38c106940 2023-07-20T02:44:42.672 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.060+0000 7fe390f6b700 1 -- 172.21.15.7:0/2870145156 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe3880042c0 con 0x7fe38c106940 2023-07-20T02:44:42.672 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.060+0000 7fe390f6b700 1 -- 172.21.15.7:0/2870145156 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe3880119b0 con 0x7fe38c106940 2023-07-20T02:44:42.672 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.061+0000 7fe3941d1700 1 -- 172.21.15.7:0/2870145156 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe38c106940 msgr2=0x7fe38c108d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:42.672 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.061+0000 7fe3941d1700 1 --2- 172.21.15.7:0/2870145156 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe38c106940 0x7fe38c108d30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe3880098a0 tx=0x7fe388009bb0 comp rx=0 tx=0).stop 2023-07-20T02:44:42.672 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.061+0000 7fe3941d1700 1 -- 172.21.15.7:0/2870145156 shutdown_connections 2023-07-20T02:44:42.672 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.061+0000 7fe3941d1700 1 --2- 172.21.15.7:0/2870145156 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe38c106940 0x7fe38c108d30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:42.673 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.061+0000 7fe3941d1700 1 -- 172.21.15.7:0/2870145156 >> 172.21.15.7:0/2870145156 conn(0x7fe38c100330 msgr2=0x7fe38c102750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:42.673 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.061+0000 7fe3941d1700 1 -- 172.21.15.7:0/2870145156 shutdown_connections 2023-07-20T02:44:42.673 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.061+0000 7fe3941d1700 1 -- 172.21.15.7:0/2870145156 wait complete. 2023-07-20T02:44:42.673 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.062+0000 7fe3941d1700 1 Processor -- start 2023-07-20T02:44:42.673 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.062+0000 7fe3941d1700 1 -- start start 2023-07-20T02:44:42.673 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.063+0000 7fe3941d1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe38c106940 0x7fe38c1062a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:42.674 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.063+0000 7fe3941d1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38c1067e0 con 0x7fe38c106940 2023-07-20T02:44:42.674 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.063+0000 7fe391f6d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe38c106940 0x7fe38c1062a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:42.674 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.063+0000 7fe391f6d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe38c106940 0x7fe38c1062a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49676/0 (socket says 172.21.15.7:49676) 2023-07-20T02:44:42.674 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.063+0000 7fe391f6d700 1 -- 172.21.15.7:0/3795504257 learned_addr learned my addr 172.21.15.7:0/3795504257 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:42.674 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.063+0000 7fe391f6d700 1 -- 172.21.15.7:0/3795504257 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe388009550 con 0x7fe38c106940 2023-07-20T02:44:42.674 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.064+0000 7fe391f6d700 1 --2- 172.21.15.7:0/3795504257 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe38c106940 0x7fe38c1062a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe388006af0 tx=0x7fe388011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:42.674 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.064+0000 7fe382ffd700 1 -- 172.21.15.7:0/3795504257 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe388003a40 con 0x7fe38c106940 2023-07-20T02:44:42.675 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.064+0000 7fe382ffd700 1 -- 172.21.15.7:0/3795504257 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe388003ba0 con 0x7fe38c106940 2023-07-20T02:44:42.675 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.064+0000 7fe382ffd700 1 -- 172.21.15.7:0/3795504257 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe3880196c0 con 0x7fe38c106940 2023-07-20T02:44:42.675 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.064+0000 7fe3941d1700 1 -- 172.21.15.7:0/3795504257 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe38c1048f0 con 0x7fe38c106940 2023-07-20T02:44:42.675 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.064+0000 7fe3941d1700 1 -- 172.21.15.7:0/3795504257 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe38c104d10 con 0x7fe38c106940 2023-07-20T02:44:42.675 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.065+0000 7fe382ffd700 1 -- 172.21.15.7:0/3795504257 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fe38801a430 con 0x7fe38c106940 2023-07-20T02:44:42.675 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.065+0000 7fe3941d1700 1 -- 172.21.15.7:0/3795504257 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe38c061da0 con 0x7fe38c106940 2023-07-20T02:44:42.675 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.066+0000 7fe382ffd700 1 --2- 172.21.15.7:0/3795504257 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7fe378038710 0x7fe37803abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:42.676 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.066+0000 7fe382ffd700 1 -- 172.21.15.7:0/3795504257 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fe38804c900 con 0x7fe38c106940 2023-07-20T02:44:42.676 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.066+0000 7fe39176c700 1 --2- 172.21.15.7:0/3795504257 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7fe378038710 0x7fe37803abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:42.676 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.067+0000 7fe39176c700 1 --2- 172.21.15.7:0/3795504257 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7fe378038710 0x7fe37803abd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe37c006fd0 tx=0x7fe37c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:42.676 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.071+0000 7fe382ffd700 1 -- 172.21.15.7:0/3795504257 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe388018350 con 0x7fe38c106940 2023-07-20T02:44:42.676 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.278+0000 7fe3941d1700 1 -- 172.21.15.7:0/3795504257 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fe38c105010 con 0x7fe38c106940 2023-07-20T02:44:42.677 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.281+0000 7fe382ffd700 1 -- 172.21.15.7:0/3795504257 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fe38804c2d0 con 0x7fe38c106940 2023-07-20T02:44:42.677 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.284+0000 7fe3941d1700 1 -- 172.21.15.7:0/3795504257 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7fe378038710 msgr2=0x7fe37803abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:42.677 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.284+0000 7fe3941d1700 1 --2- 172.21.15.7:0/3795504257 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7fe378038710 0x7fe37803abd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe37c006fd0 tx=0x7fe37c006e40 comp rx=0 tx=0).stop 2023-07-20T02:44:42.677 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.284+0000 7fe3941d1700 1 -- 172.21.15.7:0/3795504257 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe38c106940 msgr2=0x7fe38c1062a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:42.677 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.284+0000 7fe3941d1700 1 --2- 172.21.15.7:0/3795504257 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe38c106940 0x7fe38c1062a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe388006af0 tx=0x7fe388011a10 comp rx=0 tx=0).stop 2023-07-20T02:44:42.677 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.284+0000 7fe3941d1700 1 -- 172.21.15.7:0/3795504257 shutdown_connections 2023-07-20T02:44:42.677 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.284+0000 7fe3941d1700 1 --2- 172.21.15.7:0/3795504257 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7fe378038710 0x7fe37803abd0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:42.678 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.284+0000 7fe3941d1700 1 --2- 172.21.15.7:0/3795504257 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe38c106940 0x7fe38c1062a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:42.678 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.284+0000 7fe3941d1700 1 -- 172.21.15.7:0/3795504257 >> 172.21.15.7:0/3795504257 conn(0x7fe38c100330 msgr2=0x7fe38c101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:42.678 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.285+0000 7fe3941d1700 1 -- 172.21.15.7:0/3795504257 shutdown_connections 2023-07-20T02:44:42.678 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:42.285+0000 7fe3941d1700 1 -- 172.21.15.7:0/3795504257 wait complete. 2023-07-20T02:44:42.678 INFO:teuthology.orchestra.run.smithi007.stdout:Enabling the dashboard module... 2023-07-20T02:44:43.192 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:42 smithi007 ceph-mon[126353]: from='mgr.14120 172.21.15.7:0/2404405010' entity='mgr.x' 2023-07-20T02:44:43.193 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:42 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/3795504257' entity='client.admin' 2023-07-20T02:44:44.798 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:44 smithi007 ceph-mon[126353]: from='mgr.14120 172.21.15.7:0/2404405010' entity='mgr.x' 2023-07-20T02:44:44.799 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:44 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/1747281715' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-07-20T02:44:44.799 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:44 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: ignoring --setuser ceph since I am not root 2023-07-20T02:44:44.799 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:44 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: ignoring --setgroup ceph since I am not root 2023-07-20T02:44:44.800 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:44 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:44.789+0000 7f6154c2c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T02:44:44.957 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.197+0000 7f8f34d31700 1 Processor -- start 2023-07-20T02:44:44.958 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.197+0000 7f8f34d31700 1 -- start start 2023-07-20T02:44:44.958 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.198+0000 7f8f34d31700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f3007be70 0x7f8f3007a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:44.958 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.198+0000 7f8f34d31700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3007a8a0 con 0x7f8f3007be70 2023-07-20T02:44:44.959 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.198+0000 7f8f2e59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f3007be70 0x7f8f3007a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:44.959 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.198+0000 7f8f2e59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f3007be70 0x7f8f3007a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49684/0 (socket says 172.21.15.7:49684) 2023-07-20T02:44:44.959 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.198+0000 7f8f2e59c700 1 -- 172.21.15.7:0/3219119696 learned_addr learned my addr 172.21.15.7:0/3219119696 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:44.960 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.199+0000 7f8f2e59c700 1 -- 172.21.15.7:0/3219119696 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f3007a9e0 con 0x7f8f3007be70 2023-07-20T02:44:44.960 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.199+0000 7f8f2e59c700 1 --2- 172.21.15.7:0/3219119696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f3007be70 0x7f8f3007a2d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8f200098a0 tx=0x7f8f20009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdd8c574cdb63f8 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:44.960 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.199+0000 7f8f2d59a700 1 -- 172.21.15.7:0/3219119696 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8f20004160 con 0x7f8f3007be70 2023-07-20T02:44:44.960 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.199+0000 7f8f2d59a700 1 -- 172.21.15.7:0/3219119696 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f200042c0 con 0x7f8f3007be70 2023-07-20T02:44:44.960 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.199+0000 7f8f2d59a700 1 -- 172.21.15.7:0/3219119696 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8f200119b0 con 0x7f8f3007be70 2023-07-20T02:44:44.961 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.200+0000 7f8f34d31700 1 -- 172.21.15.7:0/3219119696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f3007be70 msgr2=0x7f8f3007a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:44.961 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.200+0000 7f8f34d31700 1 --2- 172.21.15.7:0/3219119696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f3007be70 0x7f8f3007a2d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8f200098a0 tx=0x7f8f20009bb0 comp rx=0 tx=0).stop 2023-07-20T02:44:44.961 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.200+0000 7f8f34d31700 1 -- 172.21.15.7:0/3219119696 shutdown_connections 2023-07-20T02:44:44.961 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.200+0000 7f8f34d31700 1 --2- 172.21.15.7:0/3219119696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f3007be70 0x7f8f3007a2d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:44.961 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.200+0000 7f8f34d31700 1 -- 172.21.15.7:0/3219119696 >> 172.21.15.7:0/3219119696 conn(0x7f8f30101440 msgr2=0x7f8f30103860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:44.961 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.200+0000 7f8f34d31700 1 -- 172.21.15.7:0/3219119696 shutdown_connections 2023-07-20T02:44:44.961 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.200+0000 7f8f34d31700 1 -- 172.21.15.7:0/3219119696 wait complete. 2023-07-20T02:44:44.962 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.201+0000 7f8f34d31700 1 Processor -- start 2023-07-20T02:44:44.962 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.202+0000 7f8f34d31700 1 -- start start 2023-07-20T02:44:44.962 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.202+0000 7f8f34d31700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f3007be70 0x7f8f3019bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:44.962 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.202+0000 7f8f34d31700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3019c2f0 con 0x7f8f3007be70 2023-07-20T02:44:44.962 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.202+0000 7f8f2e59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f3007be70 0x7f8f3019bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:44.962 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.202+0000 7f8f2e59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f3007be70 0x7f8f3019bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49692/0 (socket says 172.21.15.7:49692) 2023-07-20T02:44:44.962 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.202+0000 7f8f2e59c700 1 -- 172.21.15.7:0/1747281715 learned_addr learned my addr 172.21.15.7:0/1747281715 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:44.963 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.202+0000 7f8f2e59c700 1 -- 172.21.15.7:0/1747281715 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f20009550 con 0x7f8f3007be70 2023-07-20T02:44:44.963 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.203+0000 7f8f2e59c700 1 --2- 172.21.15.7:0/1747281715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f3007be70 0x7f8f3019bdb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8f20000c00 tx=0x7f8f20011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:44.963 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.203+0000 7f8f1f7fe700 1 -- 172.21.15.7:0/1747281715 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8f200039f0 con 0x7f8f3007be70 2023-07-20T02:44:44.964 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.203+0000 7f8f34d31700 1 -- 172.21.15.7:0/1747281715 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f3019c4f0 con 0x7f8f3007be70 2023-07-20T02:44:44.964 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.203+0000 7f8f1f7fe700 1 -- 172.21.15.7:0/1747281715 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f20003b50 con 0x7f8f3007be70 2023-07-20T02:44:44.964 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.203+0000 7f8f1f7fe700 1 -- 172.21.15.7:0/1747281715 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8f2001a780 con 0x7f8f3007be70 2023-07-20T02:44:44.964 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.203+0000 7f8f34d31700 1 -- 172.21.15.7:0/1747281715 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f3019c910 con 0x7f8f3007be70 2023-07-20T02:44:44.964 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.204+0000 7f8f34d31700 1 -- 172.21.15.7:0/1747281715 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f30195e60 con 0x7f8f3007be70 2023-07-20T02:44:44.965 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.205+0000 7f8f1f7fe700 1 -- 172.21.15.7:0/1747281715 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f8f20024440 con 0x7f8f3007be70 2023-07-20T02:44:44.965 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.205+0000 7f8f1f7fe700 1 --2- 172.21.15.7:0/1747281715 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f8f18038710 0x7f8f1803abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:44.965 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.205+0000 7f8f1f7fe700 1 -- 172.21.15.7:0/1747281715 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f8f2004c850 con 0x7f8f3007be70 2023-07-20T02:44:44.965 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.206+0000 7f8f2dd9b700 1 --2- 172.21.15.7:0/1747281715 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f8f18038710 0x7f8f1803abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:44.965 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.206+0000 7f8f2dd9b700 1 --2- 172.21.15.7:0/1747281715 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f8f18038710 0x7f8f1803abd0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8f24006fd0 tx=0x7f8f24006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:44.965 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.210+0000 7f8f1f7fe700 1 -- 172.21.15.7:0/1747281715 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8f20018b80 con 0x7f8f3007be70 2023-07-20T02:44:44.965 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:43.478+0000 7f8f34d31700 1 -- 172.21.15.7:0/1747281715 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f8f3002c970 con 0x7f8f3007be70 2023-07-20T02:44:44.966 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:44.447+0000 7f8f1f7fe700 1 -- 172.21.15.7:0/1747281715 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f8f2004ecd0 con 0x7f8f3007be70 2023-07-20T02:44:44.966 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:44.449+0000 7f8f1f7fe700 1 -- 172.21.15.7:0/1747281715 <== mon.0 v2:172.21.15.7:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f8f2004c1a0 con 0x7f8f3007be70 2023-07-20T02:44:44.966 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:44.452+0000 7f8f34d31700 1 -- 172.21.15.7:0/1747281715 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f8f18038710 msgr2=0x7f8f1803abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:44.966 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:44.452+0000 7f8f34d31700 1 --2- 172.21.15.7:0/1747281715 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f8f18038710 0x7f8f1803abd0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8f24006fd0 tx=0x7f8f24006e40 comp rx=0 tx=0).stop 2023-07-20T02:44:44.966 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:44.452+0000 7f8f34d31700 1 -- 172.21.15.7:0/1747281715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f3007be70 msgr2=0x7f8f3019bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:44.966 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:44.452+0000 7f8f34d31700 1 --2- 172.21.15.7:0/1747281715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f3007be70 0x7f8f3019bdb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8f20000c00 tx=0x7f8f20011a30 comp rx=0 tx=0).stop 2023-07-20T02:44:44.967 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:44.453+0000 7f8f34d31700 1 -- 172.21.15.7:0/1747281715 shutdown_connections 2023-07-20T02:44:44.967 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:44.453+0000 7f8f34d31700 1 --2- 172.21.15.7:0/1747281715 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f8f18038710 0x7f8f1803abd0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:44.967 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:44.453+0000 7f8f34d31700 1 --2- 172.21.15.7:0/1747281715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f3007be70 0x7f8f3019bdb0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:44.967 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:44.453+0000 7f8f34d31700 1 -- 172.21.15.7:0/1747281715 >> 172.21.15.7:0/1747281715 conn(0x7f8f30101440 msgr2=0x7f8f30102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:44.967 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:44.453+0000 7f8f34d31700 1 -- 172.21.15.7:0/1747281715 shutdown_connections 2023-07-20T02:44:44.967 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:44.453+0000 7f8f34d31700 1 -- 172.21.15.7:0/1747281715 wait complete. 2023-07-20T02:44:45.055 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:44 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:44.903+0000 7f6154c2c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T02:44:45.432 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:45 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:45.056+0000 7f6154c2c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T02:44:45.432 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:45 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:45.185+0000 7f6154c2c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T02:44:45.433 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:45 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:45.424+0000 7f6154c2c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T02:44:45.693 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:45 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/1747281715' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-07-20T02:44:45.693 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:45 smithi007 ceph-mon[126353]: mgrmap e9: x(active, since 18s) 2023-07-20T02:44:46.153 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:45 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:45.850+0000 7f6154c2c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T02:44:46.153 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:45 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:45.972+0000 7f6154c2c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T02:44:46.442 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:46 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:46.145+0000 7f6154c2c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T02:44:46.443 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:46 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:46.291+0000 7f6154c2c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T02:44:46.463 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2023-07-20T02:44:46.463 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-07-20T02:44:46.463 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-20T02:44:46.463 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-07-20T02:44:46.463 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-07-20T02:44:46.464 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2023-07-20T02:44:46.464 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.776+0000 7eff08f9e700 1 Processor -- start 2023-07-20T02:44:46.464 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.776+0000 7eff08f9e700 1 -- start start 2023-07-20T02:44:46.468 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.776+0000 7eff08f9e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff0407be70 0x7eff0407a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:46.468 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.776+0000 7eff08f9e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0407a8a0 con 0x7eff0407be70 2023-07-20T02:44:46.468 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.777+0000 7eff0259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff0407be70 0x7eff0407a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:46.468 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.777+0000 7eff0259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff0407be70 0x7eff0407a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49708/0 (socket says 172.21.15.7:49708) 2023-07-20T02:44:46.468 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.777+0000 7eff0259c700 1 -- 172.21.15.7:0/691867010 learned_addr learned my addr 172.21.15.7:0/691867010 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:46.468 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.777+0000 7eff0259c700 1 -- 172.21.15.7:0/691867010 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff0407a9e0 con 0x7eff0407be70 2023-07-20T02:44:46.469 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.777+0000 7eff0259c700 1 --2- 172.21.15.7:0/691867010 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff0407be70 0x7eff0407a2d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efef40098a0 tx=0x7efef4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8efbc0e5287d4f67 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:46.469 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.778+0000 7eff0159a700 1 -- 172.21.15.7:0/691867010 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efef4004160 con 0x7eff0407be70 2023-07-20T02:44:46.469 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.778+0000 7eff0159a700 1 -- 172.21.15.7:0/691867010 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efef40042c0 con 0x7eff0407be70 2023-07-20T02:44:46.469 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.778+0000 7eff0159a700 1 -- 172.21.15.7:0/691867010 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efef40119b0 con 0x7eff0407be70 2023-07-20T02:44:46.469 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.779+0000 7eff08f9e700 1 -- 172.21.15.7:0/691867010 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff0407be70 msgr2=0x7eff0407a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:46.469 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.779+0000 7eff08f9e700 1 --2- 172.21.15.7:0/691867010 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff0407be70 0x7eff0407a2d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efef40098a0 tx=0x7efef4009bb0 comp rx=0 tx=0).stop 2023-07-20T02:44:46.469 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.779+0000 7eff08f9e700 1 -- 172.21.15.7:0/691867010 shutdown_connections 2023-07-20T02:44:46.470 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.779+0000 7eff08f9e700 1 --2- 172.21.15.7:0/691867010 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff0407be70 0x7eff0407a2d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:46.470 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.779+0000 7eff08f9e700 1 -- 172.21.15.7:0/691867010 >> 172.21.15.7:0/691867010 conn(0x7eff04101440 msgr2=0x7eff04103860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:46.470 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.779+0000 7eff08f9e700 1 -- 172.21.15.7:0/691867010 shutdown_connections 2023-07-20T02:44:46.470 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.779+0000 7eff08f9e700 1 -- 172.21.15.7:0/691867010 wait complete. 2023-07-20T02:44:46.470 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.780+0000 7eff08f9e700 1 Processor -- start 2023-07-20T02:44:46.470 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.781+0000 7eff08f9e700 1 -- start start 2023-07-20T02:44:46.470 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.781+0000 7eff08f9e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff0407be70 0x7eff0419bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:46.470 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.781+0000 7eff08f9e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0419c2e0 con 0x7eff0407be70 2023-07-20T02:44:46.471 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.781+0000 7eff0259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff0407be70 0x7eff0419bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:46.471 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.781+0000 7eff0259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff0407be70 0x7eff0419bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49718/0 (socket says 172.21.15.7:49718) 2023-07-20T02:44:46.471 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.781+0000 7eff0259c700 1 -- 172.21.15.7:0/395770696 learned_addr learned my addr 172.21.15.7:0/395770696 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:46.471 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.782+0000 7eff0259c700 1 -- 172.21.15.7:0/395770696 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef4009550 con 0x7eff0407be70 2023-07-20T02:44:46.471 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.782+0000 7eff0259c700 1 --2- 172.21.15.7:0/395770696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff0407be70 0x7eff0419bda0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efef4000c00 tx=0x7efef4011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:46.471 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.782+0000 7efef37fe700 1 -- 172.21.15.7:0/395770696 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efef40039f0 con 0x7eff0407be70 2023-07-20T02:44:46.471 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.782+0000 7eff08f9e700 1 -- 172.21.15.7:0/395770696 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff0419c4e0 con 0x7eff0407be70 2023-07-20T02:44:46.471 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.782+0000 7efef37fe700 1 -- 172.21.15.7:0/395770696 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efef4003b50 con 0x7eff0407be70 2023-07-20T02:44:46.472 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.782+0000 7efef37fe700 1 -- 172.21.15.7:0/395770696 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efef401a780 con 0x7eff0407be70 2023-07-20T02:44:46.472 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.783+0000 7eff08f9e700 1 -- 172.21.15.7:0/395770696 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff0419c900 con 0x7eff0407be70 2023-07-20T02:44:46.472 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.784+0000 7eff08f9e700 1 -- 172.21.15.7:0/395770696 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff04195e60 con 0x7eff0407be70 2023-07-20T02:44:46.472 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.784+0000 7efef37fe700 1 -- 172.21.15.7:0/395770696 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7efef4024440 con 0x7eff0407be70 2023-07-20T02:44:46.472 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.784+0000 7efef37fe700 1 --2- 172.21.15.7:0/395770696 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7efeec038760 0x7efeec03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:46.472 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.784+0000 7efef37fe700 1 -- 172.21.15.7:0/395770696 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7efef404c7d0 con 0x7eff0407be70 2023-07-20T02:44:46.472 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.784+0000 7eff01d9b700 1 -- 172.21.15.7:0/395770696 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7efeec038760 msgr2=0x7efeec03ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/2926310425 2023-07-20T02:44:46.473 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.784+0000 7eff01d9b700 1 --2- 172.21.15.7:0/395770696 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7efeec038760 0x7efeec03ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T02:44:46.473 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.789+0000 7efef37fe700 1 -- 172.21.15.7:0/395770696 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7efef4018b80 con 0x7eff0407be70 2023-07-20T02:44:46.473 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.985+0000 7eff01d9b700 1 -- 172.21.15.7:0/395770696 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7efeec038760 msgr2=0x7efeec03ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/2926310425 2023-07-20T02:44:46.473 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:45.985+0000 7eff01d9b700 1 --2- 172.21.15.7:0/395770696 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7efeec038760 0x7efeec03ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-20T02:44:46.473 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:46.049+0000 7eff08f9e700 1 -- 172.21.15.7:0/395770696 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7eff04061380 con 0x7eff0407be70 2023-07-20T02:44:46.473 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:46.049+0000 7efef37fe700 1 -- 172.21.15.7:0/395770696 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7efef4018350 con 0x7eff0407be70 2023-07-20T02:44:46.473 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:46.051+0000 7eff08f9e700 1 -- 172.21.15.7:0/395770696 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7efeec038760 msgr2=0x7efeec03ac20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T02:44:46.474 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:46.052+0000 7eff08f9e700 1 --2- 172.21.15.7:0/395770696 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7efeec038760 0x7efeec03ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:46.474 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:46.052+0000 7eff08f9e700 1 -- 172.21.15.7:0/395770696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff0407be70 msgr2=0x7eff0419bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:46.474 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:46.052+0000 7eff08f9e700 1 --2- 172.21.15.7:0/395770696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff0407be70 0x7eff0419bda0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efef4000c00 tx=0x7efef4011a30 comp rx=0 tx=0).stop 2023-07-20T02:44:46.474 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:46.052+0000 7eff08f9e700 1 -- 172.21.15.7:0/395770696 shutdown_connections 2023-07-20T02:44:46.474 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:46.052+0000 7eff08f9e700 1 --2- 172.21.15.7:0/395770696 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7efeec038760 0x7efeec03ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:46.474 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:46.052+0000 7eff08f9e700 1 --2- 172.21.15.7:0/395770696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff0407be70 0x7eff0419bda0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:46.474 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:46.052+0000 7eff08f9e700 1 -- 172.21.15.7:0/395770696 >> 172.21.15.7:0/395770696 conn(0x7eff04101440 msgr2=0x7eff04102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:46.474 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:46.052+0000 7eff08f9e700 1 -- 172.21.15.7:0/395770696 shutdown_connections 2023-07-20T02:44:46.475 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:46.052+0000 7eff08f9e700 1 -- 172.21.15.7:0/395770696 wait complete. 2023-07-20T02:44:46.475 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for the mgr to restart... 2023-07-20T02:44:46.475 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mgr epoch 9... 2023-07-20T02:44:46.790 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:46 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/395770696' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-20T02:44:47.192 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:46 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:46.899+0000 7f6154c2c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T02:44:48.192 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:47 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:47.685+0000 7f6154c2c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T02:44:48.193 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:47 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:47.899+0000 7f6154c2c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T02:44:49.193 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:48 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:48.896+0000 7f6154c2c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T02:44:49.193 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:49 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:49.021+0000 7f6154c2c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T02:44:49.643 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:49 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:49.255+0000 7f6154c2c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T02:44:49.930 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:49 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:49.635+0000 7f6154c2c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T02:44:50.193 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:49 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:49.922+0000 7f6154c2c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T02:44:50.193 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:50 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:50.124+0000 7f6154c2c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T02:44:50.942 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:50 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:50.482+0000 7f6154c2c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T02:44:51.905 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:51 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:51.458+0000 7f6154c2c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T02:44:51.905 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:51 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:51.640+0000 7f6154c2c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T02:44:51.906 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:51 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:51.897+0000 7f6154c2c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T02:44:52.193 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:52 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:52.050+0000 7f6154c2c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T02:44:52.692 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:52 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:52.340+0000 7f6154c2c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T02:44:53.943 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:44:53 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:44:53.455+0000 7f6154c2c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T02:44:53.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:53 smithi007 ceph-mon[126353]: Active manager daemon x restarted 2023-07-20T02:44:53.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:53 smithi007 ceph-mon[126353]: Activating manager daemon x 2023-07-20T02:44:53.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:53 smithi007 ceph-mon[126353]: osdmap e3: 0 total, 0 up, 0 in 2023-07-20T02:44:53.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:53 smithi007 ceph-mon[126353]: mgrmap e10: x(active, starting, since 0.00392751s) 2023-07-20T02:44:53.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:53 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:44:53.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:53 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-20T02:44:53.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:53 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-20T02:44:53.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:53 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-20T02:44:53.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:53 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-20T02:44:53.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:53 smithi007 ceph-mon[126353]: Manager daemon x is now available 2023-07-20T02:44:53.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:53 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:44:53.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:53 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:44:54.888 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2023-07-20T02:44:54.888 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-07-20T02:44:54.888 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "initialized": true 2023-07-20T02:44:54.889 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2023-07-20T02:44:54.889 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.073+0000 7f2e9d3db700 1 Processor -- start 2023-07-20T02:44:54.889 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.073+0000 7f2e9d3db700 1 -- start start 2023-07-20T02:44:54.889 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.073+0000 7f2e9d3db700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e98105e60 0x7f2e98106280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:54.889 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.073+0000 7f2e9d3db700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e981067c0 con 0x7f2e98105e60 2023-07-20T02:44:54.889 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.073+0000 7f2e96ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e98105e60 0x7f2e98106280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:54.890 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.074+0000 7f2e96ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e98105e60 0x7f2e98106280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49726/0 (socket says 172.21.15.7:49726) 2023-07-20T02:44:54.890 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.074+0000 7f2e96ffd700 1 -- 172.21.15.7:0/3168309961 learned_addr learned my addr 172.21.15.7:0/3168309961 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:54.890 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.074+0000 7f2e96ffd700 1 -- 172.21.15.7:0/3168309961 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e98106900 con 0x7f2e98105e60 2023-07-20T02:44:54.890 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.074+0000 7f2e96ffd700 1 --2- 172.21.15.7:0/3168309961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e98105e60 0x7f2e98106280 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2e880098a0 tx=0x7f2e88009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46c1a2900a29fccd server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:54.891 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.074+0000 7f2e95ffb700 1 -- 172.21.15.7:0/3168309961 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e88004160 con 0x7f2e98105e60 2023-07-20T02:44:54.891 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.074+0000 7f2e95ffb700 1 -- 172.21.15.7:0/3168309961 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2e880042c0 con 0x7f2e98105e60 2023-07-20T02:44:54.891 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.074+0000 7f2e95ffb700 1 -- 172.21.15.7:0/3168309961 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e880119b0 con 0x7f2e98105e60 2023-07-20T02:44:54.891 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.075+0000 7f2e9d3db700 1 -- 172.21.15.7:0/3168309961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e98105e60 msgr2=0x7f2e98106280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:54.891 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.075+0000 7f2e9d3db700 1 --2- 172.21.15.7:0/3168309961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e98105e60 0x7f2e98106280 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2e880098a0 tx=0x7f2e88009bb0 comp rx=0 tx=0).stop 2023-07-20T02:44:54.891 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.075+0000 7f2e9d3db700 1 -- 172.21.15.7:0/3168309961 shutdown_connections 2023-07-20T02:44:54.891 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.075+0000 7f2e9d3db700 1 --2- 172.21.15.7:0/3168309961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e98105e60 0x7f2e98106280 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:54.892 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.075+0000 7f2e9d3db700 1 -- 172.21.15.7:0/3168309961 >> 172.21.15.7:0/3168309961 conn(0x7f2e98101440 msgr2=0x7f2e98103860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:54.892 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.075+0000 7f2e9d3db700 1 -- 172.21.15.7:0/3168309961 shutdown_connections 2023-07-20T02:44:54.892 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.075+0000 7f2e9d3db700 1 -- 172.21.15.7:0/3168309961 wait complete. 2023-07-20T02:44:54.892 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.076+0000 7f2e9d3db700 1 Processor -- start 2023-07-20T02:44:54.892 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.076+0000 7f2e9d3db700 1 -- start start 2023-07-20T02:44:54.893 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.077+0000 7f2e9d3db700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e98105e60 0x7f2e98199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:54.893 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.077+0000 7f2e9d3db700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9819a070 con 0x7f2e98105e60 2023-07-20T02:44:54.893 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.077+0000 7f2e96ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e98105e60 0x7f2e98199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:54.893 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.077+0000 7f2e96ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e98105e60 0x7f2e98199b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49728/0 (socket says 172.21.15.7:49728) 2023-07-20T02:44:54.893 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.077+0000 7f2e96ffd700 1 -- 172.21.15.7:0/824478068 learned_addr learned my addr 172.21.15.7:0/824478068 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:54.893 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.077+0000 7f2e96ffd700 1 -- 172.21.15.7:0/824478068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e88009550 con 0x7f2e98105e60 2023-07-20T02:44:54.893 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.078+0000 7f2e96ffd700 1 --2- 172.21.15.7:0/824478068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e98105e60 0x7f2e98199b30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2e88011980 tx=0x7f2e88011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:54.894 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.078+0000 7f2e7ffff700 1 -- 172.21.15.7:0/824478068 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e88003a60 con 0x7f2e98105e60 2023-07-20T02:44:54.894 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.078+0000 7f2e7ffff700 1 -- 172.21.15.7:0/824478068 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2e88003bc0 con 0x7f2e98105e60 2023-07-20T02:44:54.894 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.078+0000 7f2e9d3db700 1 -- 172.21.15.7:0/824478068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e9819a270 con 0x7f2e98105e60 2023-07-20T02:44:54.894 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.078+0000 7f2e7ffff700 1 -- 172.21.15.7:0/824478068 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e8801a670 con 0x7f2e98105e60 2023-07-20T02:44:54.894 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.078+0000 7f2e9d3db700 1 -- 172.21.15.7:0/824478068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e9819a690 con 0x7f2e98105e60 2023-07-20T02:44:54.895 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.080+0000 7f2e7ffff700 1 -- 172.21.15.7:0/824478068 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f2e88024a00 con 0x7f2e98105e60 2023-07-20T02:44:54.895 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.080+0000 7f2e7ffff700 1 --2- 172.21.15.7:0/824478068 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f2e800387b0 0x7f2e8003ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:54.895 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.080+0000 7f2e7ffff700 1 -- 172.21.15.7:0/824478068 --> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2e8003b380 con 0x7f2e800387b0 2023-07-20T02:44:54.895 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.080+0000 7f2e7ffff700 1 -- 172.21.15.7:0/824478068 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f2e8804c930 con 0x7f2e98105e60 2023-07-20T02:44:54.895 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.080+0000 7f2e967fc700 1 -- 172.21.15.7:0/824478068 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f2e800387b0 msgr2=0x7f2e8003ac70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/2926310425 2023-07-20T02:44:54.895 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.080+0000 7f2e967fc700 1 --2- 172.21.15.7:0/824478068 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f2e800387b0 0x7f2e8003ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T02:44:54.895 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.281+0000 7f2e967fc700 1 -- 172.21.15.7:0/824478068 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f2e800387b0 msgr2=0x7f2e8003ac70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/2926310425 2023-07-20T02:44:54.896 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.281+0000 7f2e967fc700 1 --2- 172.21.15.7:0/824478068 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f2e800387b0 0x7f2e8003ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-20T02:44:54.896 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.682+0000 7f2e967fc700 1 -- 172.21.15.7:0/824478068 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f2e800387b0 msgr2=0x7f2e8003ac70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/2926310425 2023-07-20T02:44:54.896 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:47.682+0000 7f2e967fc700 1 --2- 172.21.15.7:0/824478068 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f2e800387b0 0x7f2e8003ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-20T02:44:54.896 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:48.483+0000 7f2e967fc700 1 -- 172.21.15.7:0/824478068 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f2e800387b0 msgr2=0x7f2e8003ac70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/2926310425 2023-07-20T02:44:54.896 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:48.483+0000 7f2e967fc700 1 --2- 172.21.15.7:0/824478068 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f2e800387b0 0x7f2e8003ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-20T02:44:54.896 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:50.084+0000 7f2e967fc700 1 -- 172.21.15.7:0/824478068 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f2e800387b0 msgr2=0x7f2e8003ac70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/2926310425 2023-07-20T02:44:54.897 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:50.084+0000 7f2e967fc700 1 --2- 172.21.15.7:0/824478068 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f2e800387b0 0x7f2e8003ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-07-20T02:44:54.897 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:53.288+0000 7f2e967fc700 1 -- 172.21.15.7:0/824478068 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f2e800387b0 msgr2=0x7f2e8003ac70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/2926310425 2023-07-20T02:44:54.897 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:53.288+0000 7f2e967fc700 1 --2- 172.21.15.7:0/824478068 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f2e800387b0 0x7f2e8003ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-07-20T02:44:54.897 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:53.460+0000 7f2e7ffff700 1 -- 172.21.15.7:0/824478068 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mgrmap(e 10) v1 ==== 45263+0+0 (secure 0 0 0) 0x7f2e88024620 con 0x7f2e98105e60 2023-07-20T02:44:54.897 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:53.460+0000 7f2e7ffff700 1 -- 172.21.15.7:0/824478068 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f2e800387b0 msgr2=0x7f2e8003ac70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T02:44:54.897 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:53.460+0000 7f2e7ffff700 1 --2- 172.21.15.7:0/824478068 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f2e800387b0 0x7f2e8003ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:54.897 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:54.462+0000 7f2e7ffff700 1 -- 172.21.15.7:0/824478068 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mgrmap(e 11) v1 ==== 45390+0+0 (secure 0 0 0) 0x7f2e8804e250 con 0x7f2e98105e60 2023-07-20T02:44:54.897 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:54.462+0000 7f2e7ffff700 1 --2- 172.21.15.7:0/824478068 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f2e8003bfd0 0x7f2e8003e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:54.898 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:54.462+0000 7f2e7ffff700 1 -- 172.21.15.7:0/824478068 --> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2e8003b380 con 0x7f2e8003bfd0 2023-07-20T02:44:54.898 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:54.463+0000 7f2e967fc700 1 --2- 172.21.15.7:0/824478068 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f2e8003bfd0 0x7f2e8003e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:54.898 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:54.464+0000 7f2e967fc700 1 --2- 172.21.15.7:0/824478068 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f2e8003bfd0 0x7f2e8003e3c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c003a10 tx=0x7f2e8c009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:54.898 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:54.465+0000 7f2e7ffff700 1 -- 172.21.15.7:0/824478068 <== mgr.14152 v2:172.21.15.7:6800/1354542137 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f2e8003b380 con 0x7f2e8003bfd0 2023-07-20T02:44:54.898 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:54.477+0000 7f2e9d3db700 1 -- 172.21.15.7:0/824478068 --> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f2e9819ae70 con 0x7f2e8003bfd0 2023-07-20T02:44:54.898 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:54.477+0000 7f2e7ffff700 1 -- 172.21.15.7:0/824478068 <== mgr.14152 v2:172.21.15.7:6800/1354542137 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f2e9819ae70 con 0x7f2e8003bfd0 2023-07-20T02:44:54.898 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:54.478+0000 7f2e9d3db700 1 -- 172.21.15.7:0/824478068 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f2e8003bfd0 msgr2=0x7f2e8003e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:54.899 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:54.478+0000 7f2e9d3db700 1 --2- 172.21.15.7:0/824478068 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f2e8003bfd0 0x7f2e8003e3c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c003a10 tx=0x7f2e8c009270 comp rx=0 tx=0).stop 2023-07-20T02:44:54.899 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:54.478+0000 7f2e9d3db700 1 -- 172.21.15.7:0/824478068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e98105e60 msgr2=0x7f2e98199b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:54.899 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:54.478+0000 7f2e9d3db700 1 --2- 172.21.15.7:0/824478068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e98105e60 0x7f2e98199b30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2e88011980 tx=0x7f2e88011a60 comp rx=0 tx=0).stop 2023-07-20T02:44:54.899 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:54.478+0000 7f2e9d3db700 1 -- 172.21.15.7:0/824478068 shutdown_connections 2023-07-20T02:44:54.899 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:54.478+0000 7f2e9d3db700 1 --2- 172.21.15.7:0/824478068 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f2e8003bfd0 0x7f2e8003e3c0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:54.899 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:54.478+0000 7f2e9d3db700 1 --2- 172.21.15.7:0/824478068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e98105e60 0x7f2e98199b30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:54.899 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:54.478+0000 7f2e9d3db700 1 --2- 172.21.15.7:0/824478068 >> [v2:172.21.15.7:6800/2926310425,v1:172.21.15.7:6801/2926310425] conn(0x7f2e800387b0 0x7f2e8003ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:54.899 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:54.478+0000 7f2e9d3db700 1 -- 172.21.15.7:0/824478068 >> 172.21.15.7:0/824478068 conn(0x7f2e98101440 msgr2=0x7f2e98102b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:54.900 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:54.478+0000 7f2e9d3db700 1 -- 172.21.15.7:0/824478068 shutdown_connections 2023-07-20T02:44:54.900 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:54.478+0000 7f2e9d3db700 1 -- 172.21.15.7:0/824478068 wait complete. 2023-07-20T02:44:54.900 INFO:teuthology.orchestra.run.smithi007.stdout:mgr epoch 9 is available 2023-07-20T02:44:54.900 INFO:teuthology.orchestra.run.smithi007.stdout:Generating a dashboard self-signed certificate... 2023-07-20T02:44:54.943 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:54 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-20T02:44:54.943 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:54 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-20T02:44:54.943 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:54 smithi007 ceph-mon[126353]: mgrmap e11: x(active, since 1.00737s) 2023-07-20T02:44:55.884 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:55 smithi007 ceph-mon[126353]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-20T02:44:55.885 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:55 smithi007 ceph-mon[126353]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-20T02:44:56.271 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-07-20T02:44:56.271 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.479+0000 7f27fd16a700 1 Processor -- start 2023-07-20T02:44:56.271 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.480+0000 7f27fd16a700 1 -- start start 2023-07-20T02:44:56.272 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.480+0000 7f27fd16a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27f8105e60 0x7f27f8106280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:56.272 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.480+0000 7f27fd16a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f81067c0 con 0x7f27f8105e60 2023-07-20T02:44:56.272 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.481+0000 7f27f6d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27f8105e60 0x7f27f8106280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:56.272 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.481+0000 7f27f6d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27f8105e60 0x7f27f8106280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60528/0 (socket says 172.21.15.7:60528) 2023-07-20T02:44:56.272 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.481+0000 7f27f6d9d700 1 -- 172.21.15.7:0/1637078593 learned_addr learned my addr 172.21.15.7:0/1637078593 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:56.272 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.481+0000 7f27f6d9d700 1 -- 172.21.15.7:0/1637078593 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f8106900 con 0x7f27f8105e60 2023-07-20T02:44:56.272 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.482+0000 7f27f6d9d700 1 --2- 172.21.15.7:0/1637078593 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27f8105e60 0x7f27f8106280 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f27e80098a0 tx=0x7f27e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e80a90e5fb3754a3 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:56.273 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.482+0000 7f27f5d9b700 1 -- 172.21.15.7:0/1637078593 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27e8004160 con 0x7f27f8105e60 2023-07-20T02:44:56.273 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.482+0000 7f27f5d9b700 1 -- 172.21.15.7:0/1637078593 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f27e80042c0 con 0x7f27f8105e60 2023-07-20T02:44:56.273 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.482+0000 7f27f5d9b700 1 -- 172.21.15.7:0/1637078593 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27e80119b0 con 0x7f27f8105e60 2023-07-20T02:44:56.273 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.483+0000 7f27fd16a700 1 -- 172.21.15.7:0/1637078593 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27f8105e60 msgr2=0x7f27f8106280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:56.273 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.483+0000 7f27fd16a700 1 --2- 172.21.15.7:0/1637078593 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27f8105e60 0x7f27f8106280 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f27e80098a0 tx=0x7f27e8009bb0 comp rx=0 tx=0).stop 2023-07-20T02:44:56.273 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.483+0000 7f27fd16a700 1 -- 172.21.15.7:0/1637078593 shutdown_connections 2023-07-20T02:44:56.273 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.483+0000 7f27fd16a700 1 --2- 172.21.15.7:0/1637078593 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27f8105e60 0x7f27f8106280 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:56.273 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.483+0000 7f27fd16a700 1 -- 172.21.15.7:0/1637078593 >> 172.21.15.7:0/1637078593 conn(0x7f27f8101440 msgr2=0x7f27f8103860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:56.274 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.483+0000 7f27fd16a700 1 -- 172.21.15.7:0/1637078593 shutdown_connections 2023-07-20T02:44:56.274 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.483+0000 7f27fd16a700 1 -- 172.21.15.7:0/1637078593 wait complete. 2023-07-20T02:44:56.274 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.484+0000 7f27fd16a700 1 Processor -- start 2023-07-20T02:44:56.274 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.484+0000 7f27fd16a700 1 -- start start 2023-07-20T02:44:56.274 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.484+0000 7f27fd16a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27f8105e60 0x7f27f807bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:56.274 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.484+0000 7f27fd16a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f807c160 con 0x7f27f8105e60 2023-07-20T02:44:56.274 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.485+0000 7f27f6d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27f8105e60 0x7f27f807bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:56.275 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.485+0000 7f27f6d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27f8105e60 0x7f27f807bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60534/0 (socket says 172.21.15.7:60534) 2023-07-20T02:44:56.275 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.485+0000 7f27f6d9d700 1 -- 172.21.15.7:0/2027933358 learned_addr learned my addr 172.21.15.7:0/2027933358 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:56.275 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.485+0000 7f27f6d9d700 1 -- 172.21.15.7:0/2027933358 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e8009550 con 0x7f27f8105e60 2023-07-20T02:44:56.275 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.485+0000 7f27f6d9d700 1 --2- 172.21.15.7:0/2027933358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27f8105e60 0x7f27f807bc20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f27e8011980 tx=0x7f27e8011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:56.275 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.486+0000 7f27dffff700 1 -- 172.21.15.7:0/2027933358 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27e8003a60 con 0x7f27f8105e60 2023-07-20T02:44:56.275 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.486+0000 7f27fd16a700 1 -- 172.21.15.7:0/2027933358 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27f807a270 con 0x7f27f8105e60 2023-07-20T02:44:56.275 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.486+0000 7f27dffff700 1 -- 172.21.15.7:0/2027933358 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f27e8003bc0 con 0x7f27f8105e60 2023-07-20T02:44:56.275 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.486+0000 7f27dffff700 1 -- 172.21.15.7:0/2027933358 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27e801a670 con 0x7f27f8105e60 2023-07-20T02:44:56.276 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.486+0000 7f27fd16a700 1 -- 172.21.15.7:0/2027933358 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27f807a690 con 0x7f27f8105e60 2023-07-20T02:44:56.276 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.487+0000 7f27fd16a700 1 -- 172.21.15.7:0/2027933358 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27f8061da0 con 0x7f27f8105e60 2023-07-20T02:44:56.276 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.487+0000 7f27dffff700 1 -- 172.21.15.7:0/2027933358 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 11) v1 ==== 45390+0+0 (secure 0 0 0) 0x7f27e8024a00 con 0x7f27f8105e60 2023-07-20T02:44:56.276 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.488+0000 7f27dffff700 1 --2- 172.21.15.7:0/2027933358 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f27e0038630 0x7f27e003aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:56.276 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.488+0000 7f27dffff700 1 -- 172.21.15.7:0/2027933358 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f27e804c770 con 0x7f27f8105e60 2023-07-20T02:44:56.276 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.488+0000 7f27f659c700 1 --2- 172.21.15.7:0/2027933358 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f27e0038630 0x7f27e003aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:56.277 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.489+0000 7f27f659c700 1 --2- 172.21.15.7:0/2027933358 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f27e0038630 0x7f27e003aaf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f27ec006fd0 tx=0x7f27ec006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:56.277 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.493+0000 7f27dffff700 1 -- 172.21.15.7:0/2027933358 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f27e8018b80 con 0x7f27f8105e60 2023-07-20T02:44:56.277 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.514+0000 7f27dffff700 1 -- 172.21.15.7:0/2027933358 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f27e802d910 con 0x7f27f8105e60 2023-07-20T02:44:56.277 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.727+0000 7f27fd16a700 1 -- 172.21.15.7:0/2027933358 --> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f27f8103720 con 0x7f27e0038630 2023-07-20T02:44:56.277 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.845+0000 7f27dffff700 1 -- 172.21.15.7:0/2027933358 <== mgr.14152 v2:172.21.15.7:6800/1354542137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f27f8103720 con 0x7f27e0038630 2023-07-20T02:44:56.277 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.849+0000 7f27fd16a700 1 -- 172.21.15.7:0/2027933358 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f27e0038630 msgr2=0x7f27e003aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:56.277 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.849+0000 7f27fd16a700 1 --2- 172.21.15.7:0/2027933358 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f27e0038630 0x7f27e003aaf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f27ec006fd0 tx=0x7f27ec006e40 comp rx=0 tx=0).stop 2023-07-20T02:44:56.278 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.849+0000 7f27fd16a700 1 -- 172.21.15.7:0/2027933358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27f8105e60 msgr2=0x7f27f807bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:56.278 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.849+0000 7f27fd16a700 1 --2- 172.21.15.7:0/2027933358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27f8105e60 0x7f27f807bc20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f27e8011980 tx=0x7f27e8011a60 comp rx=0 tx=0).stop 2023-07-20T02:44:56.278 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.849+0000 7f27fd16a700 1 -- 172.21.15.7:0/2027933358 shutdown_connections 2023-07-20T02:44:56.278 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.849+0000 7f27fd16a700 1 --2- 172.21.15.7:0/2027933358 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f27e0038630 0x7f27e003aaf0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:56.278 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.849+0000 7f27fd16a700 1 --2- 172.21.15.7:0/2027933358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27f8105e60 0x7f27f807bc20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:56.278 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.849+0000 7f27fd16a700 1 -- 172.21.15.7:0/2027933358 >> 172.21.15.7:0/2027933358 conn(0x7f27f8101440 msgr2=0x7f27f8102b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:56.278 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.849+0000 7f27fd16a700 1 -- 172.21.15.7:0/2027933358 shutdown_connections 2023-07-20T02:44:56.278 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:55.849+0000 7f27fd16a700 1 -- 172.21.15.7:0/2027933358 wait complete. 2023-07-20T02:44:56.279 INFO:teuthology.orchestra.run.smithi007.stdout:Creating initial admin user... 2023-07-20T02:44:56.561 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:56 smithi007 ceph-mon[126353]: [20/Jul/2023:02:44:54] ENGINE Bus STARTING 2023-07-20T02:44:56.562 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:56 smithi007 ceph-mon[126353]: [20/Jul/2023:02:44:54] ENGINE Serving on https://172.21.15.7:7150 2023-07-20T02:44:56.562 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:56 smithi007 ceph-mon[126353]: [20/Jul/2023:02:44:54] ENGINE Serving on http://172.21.15.7:8765 2023-07-20T02:44:56.562 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:56 smithi007 ceph-mon[126353]: [20/Jul/2023:02:44:54] ENGINE Bus STARTED 2023-07-20T02:44:56.562 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:56 smithi007 ceph-mon[126353]: mgrmap e12: x(active, since 2s) 2023-07-20T02:44:56.562 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:56 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:44:56.562 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:56 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:44:56.563 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:56 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:44:57.693 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:57 smithi007 ceph-mon[126353]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:44:57.694 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:57 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:44:57.694 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:57 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:44:57.859 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$voVDRut8oz8okLsSNsN5.eC8opKhmGoUs1d89JjF0fu6tVNvHan0e", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1689821097, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-07-20T02:44:57.859 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.843+0000 7f57cae72700 1 Processor -- start 2023-07-20T02:44:57.859 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.843+0000 7f57cae72700 1 -- start start 2023-07-20T02:44:57.860 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.843+0000 7f57cae72700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57c4104730 0x7f57c4104b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:57.860 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.843+0000 7f57cae72700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c4079e00 con 0x7f57c4104730 2023-07-20T02:44:57.860 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.844+0000 7f57c8c0e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57c4104730 0x7f57c4104b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:57.860 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.844+0000 7f57c8c0e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57c4104730 0x7f57c4104b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60546/0 (socket says 172.21.15.7:60546) 2023-07-20T02:44:57.861 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.844+0000 7f57c8c0e700 1 -- 172.21.15.7:0/158911450 learned_addr learned my addr 172.21.15.7:0/158911450 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:57.862 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.844+0000 7f57c8c0e700 1 -- 172.21.15.7:0/158911450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c4105090 con 0x7f57c4104730 2023-07-20T02:44:57.862 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.845+0000 7f57c8c0e700 1 --2- 172.21.15.7:0/158911450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57c4104730 0x7f57c4104b50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f57b80098a0 tx=0x7f57b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcf1a9a547ae8fef server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:57.862 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.845+0000 7f57c37fe700 1 -- 172.21.15.7:0/158911450 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f57b8004160 con 0x7f57c4104730 2023-07-20T02:44:57.862 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.845+0000 7f57c37fe700 1 -- 172.21.15.7:0/158911450 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f57b80042c0 con 0x7f57c4104730 2023-07-20T02:44:57.862 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.845+0000 7f57c37fe700 1 -- 172.21.15.7:0/158911450 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f57b80119b0 con 0x7f57c4104730 2023-07-20T02:44:57.862 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.845+0000 7f57cae72700 1 -- 172.21.15.7:0/158911450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57c4104730 msgr2=0x7f57c4104b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:57.863 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.845+0000 7f57cae72700 1 --2- 172.21.15.7:0/158911450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57c4104730 0x7f57c4104b50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f57b80098a0 tx=0x7f57b8009bb0 comp rx=0 tx=0).stop 2023-07-20T02:44:57.863 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.846+0000 7f57cae72700 1 -- 172.21.15.7:0/158911450 shutdown_connections 2023-07-20T02:44:57.863 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.846+0000 7f57cae72700 1 --2- 172.21.15.7:0/158911450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57c4104730 0x7f57c4104b50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:57.863 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.846+0000 7f57cae72700 1 -- 172.21.15.7:0/158911450 >> 172.21.15.7:0/158911450 conn(0x7f57c4100330 msgr2=0x7f57c4102750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:57.863 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.846+0000 7f57cae72700 1 -- 172.21.15.7:0/158911450 shutdown_connections 2023-07-20T02:44:57.863 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.846+0000 7f57cae72700 1 -- 172.21.15.7:0/158911450 wait complete. 2023-07-20T02:44:57.864 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.848+0000 7f57cae72700 1 Processor -- start 2023-07-20T02:44:57.864 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.848+0000 7f57cae72700 1 -- start start 2023-07-20T02:44:57.864 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.848+0000 7f57cae72700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57c4104730 0x7f57c419bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:57.864 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.848+0000 7f57cae72700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c419c2e0 con 0x7f57c4104730 2023-07-20T02:44:57.864 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.848+0000 7f57c8c0e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57c4104730 0x7f57c419bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:57.864 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.849+0000 7f57c8c0e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57c4104730 0x7f57c419bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60558/0 (socket says 172.21.15.7:60558) 2023-07-20T02:44:57.865 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.849+0000 7f57c8c0e700 1 -- 172.21.15.7:0/3339037991 learned_addr learned my addr 172.21.15.7:0/3339037991 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:57.865 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.849+0000 7f57c8c0e700 1 -- 172.21.15.7:0/3339037991 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b8009550 con 0x7f57c4104730 2023-07-20T02:44:57.865 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.849+0000 7f57c8c0e700 1 --2- 172.21.15.7:0/3339037991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57c4104730 0x7f57c419bda0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f57b8006af0 tx=0x7f57b8011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:57.865 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.849+0000 7f57c1ffb700 1 -- 172.21.15.7:0/3339037991 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f57b8003a40 con 0x7f57c4104730 2023-07-20T02:44:57.865 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.849+0000 7f57c1ffb700 1 -- 172.21.15.7:0/3339037991 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f57b8003ba0 con 0x7f57c4104730 2023-07-20T02:44:57.866 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.849+0000 7f57c1ffb700 1 -- 172.21.15.7:0/3339037991 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f57b80196c0 con 0x7f57c4104730 2023-07-20T02:44:57.866 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.849+0000 7f57cae72700 1 -- 172.21.15.7:0/3339037991 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57c419c4e0 con 0x7f57c4104730 2023-07-20T02:44:57.866 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.850+0000 7f57cae72700 1 -- 172.21.15.7:0/3339037991 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c419c900 con 0x7f57c4104730 2023-07-20T02:44:57.866 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.851+0000 7f57c1ffb700 1 -- 172.21.15.7:0/3339037991 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f57b801aa80 con 0x7f57c4104730 2023-07-20T02:44:57.867 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.851+0000 7f57cae72700 1 -- 172.21.15.7:0/3339037991 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57c4195670 con 0x7f57c4104730 2023-07-20T02:44:57.867 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.851+0000 7f57c1ffb700 1 --2- 172.21.15.7:0/3339037991 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f57b4038760 0x7f57b403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:57.867 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.851+0000 7f57c1ffb700 1 -- 172.21.15.7:0/3339037991 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f57b804cad0 con 0x7f57c4104730 2023-07-20T02:44:57.868 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.852+0000 7f57c3fff700 1 --2- 172.21.15.7:0/3339037991 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f57b4038760 0x7f57b403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:57.868 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.852+0000 7f57c3fff700 1 --2- 172.21.15.7:0/3339037991 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f57b4038760 0x7f57b403ac20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f57b0006fd0 tx=0x7f57b0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:57.868 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:56.857+0000 7f57c1ffb700 1 -- 172.21.15.7:0/3339037991 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f57b801a430 con 0x7f57c4104730 2023-07-20T02:44:57.869 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:57.082+0000 7f57cae72700 1 -- 172.21.15.7:0/3339037991 --> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f57c4061bc0 con 0x7f57b4038760 2023-07-20T02:44:57.869 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:57.331+0000 7f57c1ffb700 1 -- 172.21.15.7:0/3339037991 <== mgr.14152 v2:172.21.15.7:6800/1354542137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f57c4061bc0 con 0x7f57b4038760 2023-07-20T02:44:57.869 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:57.333+0000 7f57cae72700 1 -- 172.21.15.7:0/3339037991 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f57b4038760 msgr2=0x7f57b403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:57.869 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:57.333+0000 7f57cae72700 1 --2- 172.21.15.7:0/3339037991 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f57b4038760 0x7f57b403ac20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f57b0006fd0 tx=0x7f57b0006e40 comp rx=0 tx=0).stop 2023-07-20T02:44:57.870 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:57.333+0000 7f57cae72700 1 -- 172.21.15.7:0/3339037991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57c4104730 msgr2=0x7f57c419bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:57.870 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:57.333+0000 7f57cae72700 1 --2- 172.21.15.7:0/3339037991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57c4104730 0x7f57c419bda0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f57b8006af0 tx=0x7f57b8011a10 comp rx=0 tx=0).stop 2023-07-20T02:44:57.870 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:57.334+0000 7f57cae72700 1 -- 172.21.15.7:0/3339037991 shutdown_connections 2023-07-20T02:44:57.870 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:57.334+0000 7f57cae72700 1 --2- 172.21.15.7:0/3339037991 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f57b4038760 0x7f57b403ac20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:57.871 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:57.334+0000 7f57cae72700 1 --2- 172.21.15.7:0/3339037991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57c4104730 0x7f57c419bda0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:57.871 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:57.334+0000 7f57cae72700 1 -- 172.21.15.7:0/3339037991 >> 172.21.15.7:0/3339037991 conn(0x7f57c4100330 msgr2=0x7f57c41902f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:57.871 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:57.334+0000 7f57cae72700 1 -- 172.21.15.7:0/3339037991 shutdown_connections 2023-07-20T02:44:57.871 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:57.334+0000 7f57cae72700 1 -- 172.21.15.7:0/3339037991 wait complete. 2023-07-20T02:44:57.872 INFO:teuthology.orchestra.run.smithi007.stdout:Fetching dashboard port number... 2023-07-20T02:44:58.809 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:58 smithi007 ceph-mon[126353]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:44:59.243 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 8443 2023-07-20T02:44:59.243 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.406+0000 7f09a02f5700 1 Processor -- start 2023-07-20T02:44:59.243 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.407+0000 7f09a02f5700 1 -- start start 2023-07-20T02:44:59.244 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.407+0000 7f09a02f5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0998105e60 0x7f0998106280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:59.244 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.407+0000 7f09a02f5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09981067c0 con 0x7f0998105e60 2023-07-20T02:44:59.244 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.407+0000 7f099e091700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0998105e60 0x7f0998106280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:59.244 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.408+0000 7f099e091700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0998105e60 0x7f0998106280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60564/0 (socket says 172.21.15.7:60564) 2023-07-20T02:44:59.244 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.408+0000 7f099e091700 1 -- 172.21.15.7:0/1383317381 learned_addr learned my addr 172.21.15.7:0/1383317381 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:59.244 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.408+0000 7f099e091700 1 -- 172.21.15.7:0/1383317381 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0998106900 con 0x7f0998105e60 2023-07-20T02:44:59.244 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.408+0000 7f099e091700 1 --2- 172.21.15.7:0/1383317381 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0998105e60 0x7f0998106280 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f09940098a0 tx=0x7f0994009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d795047992879e3 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:59.245 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.408+0000 7f099d08f700 1 -- 172.21.15.7:0/1383317381 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0994004160 con 0x7f0998105e60 2023-07-20T02:44:59.245 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.409+0000 7f099d08f700 1 -- 172.21.15.7:0/1383317381 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f09940042c0 con 0x7f0998105e60 2023-07-20T02:44:59.249 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.409+0000 7f099d08f700 1 -- 172.21.15.7:0/1383317381 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f09940119b0 con 0x7f0998105e60 2023-07-20T02:44:59.249 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.409+0000 7f09a02f5700 1 -- 172.21.15.7:0/1383317381 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0998105e60 msgr2=0x7f0998106280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:59.249 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.409+0000 7f09a02f5700 1 --2- 172.21.15.7:0/1383317381 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0998105e60 0x7f0998106280 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f09940098a0 tx=0x7f0994009bb0 comp rx=0 tx=0).stop 2023-07-20T02:44:59.249 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.409+0000 7f09a02f5700 1 -- 172.21.15.7:0/1383317381 shutdown_connections 2023-07-20T02:44:59.250 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.409+0000 7f09a02f5700 1 --2- 172.21.15.7:0/1383317381 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0998105e60 0x7f0998106280 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:59.250 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.409+0000 7f09a02f5700 1 -- 172.21.15.7:0/1383317381 >> 172.21.15.7:0/1383317381 conn(0x7f0998101440 msgr2=0x7f0998103860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:59.250 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.409+0000 7f09a02f5700 1 -- 172.21.15.7:0/1383317381 shutdown_connections 2023-07-20T02:44:59.250 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.409+0000 7f09a02f5700 1 -- 172.21.15.7:0/1383317381 wait complete. 2023-07-20T02:44:59.250 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.411+0000 7f09a02f5700 1 Processor -- start 2023-07-20T02:44:59.250 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.411+0000 7f09a02f5700 1 -- start start 2023-07-20T02:44:59.250 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.412+0000 7f09a02f5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0998105e60 0x7f0998199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:59.250 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.412+0000 7f09a02f5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f099819a090 con 0x7f0998105e60 2023-07-20T02:44:59.251 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.412+0000 7f099e091700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0998105e60 0x7f0998199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:59.251 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.412+0000 7f099e091700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0998105e60 0x7f0998199b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60574/0 (socket says 172.21.15.7:60574) 2023-07-20T02:44:59.251 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.412+0000 7f099e091700 1 -- 172.21.15.7:0/218353072 learned_addr learned my addr 172.21.15.7:0/218353072 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:44:59.251 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.412+0000 7f099e091700 1 -- 172.21.15.7:0/218353072 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0994009550 con 0x7f0998105e60 2023-07-20T02:44:59.251 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.413+0000 7f099e091700 1 --2- 172.21.15.7:0/218353072 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0998105e60 0x7f0998199b50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0994011980 tx=0x7f0994011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:59.251 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.413+0000 7f098f7fe700 1 -- 172.21.15.7:0/218353072 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0994003a60 con 0x7f0998105e60 2023-07-20T02:44:59.251 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.413+0000 7f098f7fe700 1 -- 172.21.15.7:0/218353072 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0994003bc0 con 0x7f0998105e60 2023-07-20T02:44:59.252 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.413+0000 7f098f7fe700 1 -- 172.21.15.7:0/218353072 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f099401a670 con 0x7f0998105e60 2023-07-20T02:44:59.252 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.413+0000 7f09a02f5700 1 -- 172.21.15.7:0/218353072 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f099819a290 con 0x7f0998105e60 2023-07-20T02:44:59.252 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.413+0000 7f09a02f5700 1 -- 172.21.15.7:0/218353072 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f099819a6b0 con 0x7f0998105e60 2023-07-20T02:44:59.252 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.414+0000 7f098f7fe700 1 -- 172.21.15.7:0/218353072 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f0994024a00 con 0x7f0998105e60 2023-07-20T02:44:59.252 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.414+0000 7f09a02f5700 1 -- 172.21.15.7:0/218353072 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0998061da0 con 0x7f0998105e60 2023-07-20T02:44:59.252 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.415+0000 7f098f7fe700 1 --2- 172.21.15.7:0/218353072 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f09840387b0 0x7f098403ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:44:59.252 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.415+0000 7f098f7fe700 1 -- 172.21.15.7:0/218353072 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f099404ca10 con 0x7f0998105e60 2023-07-20T02:44:59.253 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.415+0000 7f099d890700 1 --2- 172.21.15.7:0/218353072 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f09840387b0 0x7f098403ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:44:59.253 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.416+0000 7f099d890700 1 --2- 172.21.15.7:0/218353072 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f09840387b0 0x7f098403ac70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0988006fd0 tx=0x7f0988006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:44:59.253 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.420+0000 7f098f7fe700 1 -- 172.21.15.7:0/218353072 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0994024440 con 0x7f0998105e60 2023-07-20T02:44:59.253 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.633+0000 7f09a02f5700 1 -- 172.21.15.7:0/218353072 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f0998107cd0 con 0x7f0998105e60 2023-07-20T02:44:59.253 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.634+0000 7f098f7fe700 1 -- 172.21.15.7:0/218353072 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f0994018350 con 0x7f0998105e60 2023-07-20T02:44:59.253 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.637+0000 7f09a02f5700 1 -- 172.21.15.7:0/218353072 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f09840387b0 msgr2=0x7f098403ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:59.253 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.637+0000 7f09a02f5700 1 --2- 172.21.15.7:0/218353072 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f09840387b0 0x7f098403ac70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0988006fd0 tx=0x7f0988006e40 comp rx=0 tx=0).stop 2023-07-20T02:44:59.254 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.637+0000 7f09a02f5700 1 -- 172.21.15.7:0/218353072 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0998105e60 msgr2=0x7f0998199b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:44:59.254 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.637+0000 7f09a02f5700 1 --2- 172.21.15.7:0/218353072 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0998105e60 0x7f0998199b50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0994011980 tx=0x7f0994011a60 comp rx=0 tx=0).stop 2023-07-20T02:44:59.254 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.637+0000 7f09a02f5700 1 -- 172.21.15.7:0/218353072 shutdown_connections 2023-07-20T02:44:59.254 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.637+0000 7f09a02f5700 1 --2- 172.21.15.7:0/218353072 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f09840387b0 0x7f098403ac70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:59.254 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.637+0000 7f09a02f5700 1 --2- 172.21.15.7:0/218353072 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0998105e60 0x7f0998199b50 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:44:59.254 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.637+0000 7f09a02f5700 1 -- 172.21.15.7:0/218353072 >> 172.21.15.7:0/218353072 conn(0x7f0998101440 msgr2=0x7f0998102b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:44:59.255 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.637+0000 7f09a02f5700 1 -- 172.21.15.7:0/218353072 shutdown_connections 2023-07-20T02:44:59.255 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:58.637+0000 7f09a02f5700 1 -- 172.21.15.7:0/218353072 wait complete. 2023-07-20T02:44:59.266 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-20T02:44:59.267 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout disabled 2023-07-20T02:44:59.283 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-20T02:44:59.283 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2023-07-20T02:44:59.283 INFO:teuthology.orchestra.run.smithi007.stdout:firewalld.service is not enabled 2023-07-20T02:44:59.283 INFO:teuthology.orchestra.run.smithi007.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-07-20T02:44:59.285 INFO:teuthology.orchestra.run.smithi007.stdout:Ceph Dashboard is now available at: 2023-07-20T02:44:59.285 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:44:59.285 INFO:teuthology.orchestra.run.smithi007.stdout: URL: https://smithi007.front.sepia.ceph.com:8443/ 2023-07-20T02:44:59.286 INFO:teuthology.orchestra.run.smithi007.stdout: User: admin 2023-07-20T02:44:59.286 INFO:teuthology.orchestra.run.smithi007.stdout: Password: e76w7gexb3 2023-07-20T02:44:59.286 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:44:59.286 INFO:teuthology.orchestra.run.smithi007.stdout:Saving cluster configuration to /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/config directory 2023-07-20T02:44:59.287 INFO:teuthology.orchestra.run.smithi007.stdout:Enabling autotune for osd_memory_target 2023-07-20T02:44:59.943 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:44:59 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/218353072' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-07-20T02:45:00.686 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.977+0000 7f35a35c9700 1 Processor -- start 2023-07-20T02:45:00.686 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.977+0000 7f35a35c9700 1 -- start start 2023-07-20T02:45:00.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.978+0000 7f35a35c9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f359c105e60 0x7f359c106280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:00.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.978+0000 7f35a35c9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359c1067c0 con 0x7f359c105e60 2023-07-20T02:45:00.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.978+0000 7f35a1365700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f359c105e60 0x7f359c106280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:00.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.978+0000 7f35a1365700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f359c105e60 0x7f359c106280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60586/0 (socket says 172.21.15.7:60586) 2023-07-20T02:45:00.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.978+0000 7f35a1365700 1 -- 172.21.15.7:0/1949461685 learned_addr learned my addr 172.21.15.7:0/1949461685 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:45:00.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.978+0000 7f35a1365700 1 -- 172.21.15.7:0/1949461685 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359c106900 con 0x7f359c105e60 2023-07-20T02:45:00.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.979+0000 7f35a1365700 1 --2- 172.21.15.7:0/1949461685 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f359c105e60 0x7f359c106280 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f35980098a0 tx=0x7f3598009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=587b026e85cea0cc server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:45:00.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.979+0000 7f3593fff700 1 -- 172.21.15.7:0/1949461685 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3598004160 con 0x7f359c105e60 2023-07-20T02:45:00.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.979+0000 7f3593fff700 1 -- 172.21.15.7:0/1949461685 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f35980042c0 con 0x7f359c105e60 2023-07-20T02:45:00.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.979+0000 7f3593fff700 1 -- 172.21.15.7:0/1949461685 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f35980119b0 con 0x7f359c105e60 2023-07-20T02:45:00.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.979+0000 7f35a35c9700 1 -- 172.21.15.7:0/1949461685 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f359c105e60 msgr2=0x7f359c106280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:00.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.979+0000 7f35a35c9700 1 --2- 172.21.15.7:0/1949461685 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f359c105e60 0x7f359c106280 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f35980098a0 tx=0x7f3598009bb0 comp rx=0 tx=0).stop 2023-07-20T02:45:00.689 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.980+0000 7f35a35c9700 1 -- 172.21.15.7:0/1949461685 shutdown_connections 2023-07-20T02:45:00.689 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.980+0000 7f35a35c9700 1 --2- 172.21.15.7:0/1949461685 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f359c105e60 0x7f359c106280 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:00.689 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.980+0000 7f35a35c9700 1 -- 172.21.15.7:0/1949461685 >> 172.21.15.7:0/1949461685 conn(0x7f359c101440 msgr2=0x7f359c103860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:45:00.689 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.980+0000 7f35a35c9700 1 -- 172.21.15.7:0/1949461685 shutdown_connections 2023-07-20T02:45:00.689 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.980+0000 7f35a35c9700 1 -- 172.21.15.7:0/1949461685 wait complete. 2023-07-20T02:45:00.689 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.981+0000 7f35a35c9700 1 Processor -- start 2023-07-20T02:45:00.689 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.981+0000 7f35a35c9700 1 -- start start 2023-07-20T02:45:00.690 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.981+0000 7f35a35c9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f359c105e60 0x7f359c19bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:00.690 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.981+0000 7f35a35c9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359c19c290 con 0x7f359c105e60 2023-07-20T02:45:00.690 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.981+0000 7f35a1365700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f359c105e60 0x7f359c19bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:00.690 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.981+0000 7f35a1365700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f359c105e60 0x7f359c19bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60600/0 (socket says 172.21.15.7:60600) 2023-07-20T02:45:00.690 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.981+0000 7f35a1365700 1 -- 172.21.15.7:0/2721328961 learned_addr learned my addr 172.21.15.7:0/2721328961 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:45:00.690 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.982+0000 7f35a1365700 1 -- 172.21.15.7:0/2721328961 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3598009550 con 0x7f359c105e60 2023-07-20T02:45:00.690 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.982+0000 7f35a1365700 1 --2- 172.21.15.7:0/2721328961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f359c105e60 0x7f359c19bd50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3598011980 tx=0x7f3598011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:45:00.691 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.982+0000 7f35927fc700 1 -- 172.21.15.7:0/2721328961 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3598003a60 con 0x7f359c105e60 2023-07-20T02:45:00.691 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.982+0000 7f35a35c9700 1 -- 172.21.15.7:0/2721328961 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f359c19c490 con 0x7f359c105e60 2023-07-20T02:45:00.691 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.982+0000 7f35927fc700 1 -- 172.21.15.7:0/2721328961 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3598003bc0 con 0x7f359c105e60 2023-07-20T02:45:00.691 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.982+0000 7f35927fc700 1 -- 172.21.15.7:0/2721328961 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f359801a670 con 0x7f359c105e60 2023-07-20T02:45:00.691 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.983+0000 7f35a35c9700 1 -- 172.21.15.7:0/2721328961 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f359c19c8b0 con 0x7f359c105e60 2023-07-20T02:45:00.691 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.983+0000 7f35a35c9700 1 -- 172.21.15.7:0/2721328961 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f359c061da0 con 0x7f359c105e60 2023-07-20T02:45:00.691 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.983+0000 7f35927fc700 1 -- 172.21.15.7:0/2721328961 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f3598024a00 con 0x7f359c105e60 2023-07-20T02:45:00.692 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.984+0000 7f35927fc700 1 --2- 172.21.15.7:0/2721328961 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f3588038850 0x7f358803ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:00.692 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.984+0000 7f35927fc700 1 -- 172.21.15.7:0/2721328961 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f359804ca70 con 0x7f359c105e60 2023-07-20T02:45:00.692 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.984+0000 7f35a0b64700 1 --2- 172.21.15.7:0/2721328961 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f3588038850 0x7f358803ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:00.692 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.985+0000 7f35a0b64700 1 --2- 172.21.15.7:0/2721328961 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f3588038850 0x7f358803ad10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f358c006fd0 tx=0x7f358c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:45:00.692 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:44:59.989+0000 7f35927fc700 1 -- 172.21.15.7:0/2721328961 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3598024440 con 0x7f359c105e60 2023-07-20T02:45:00.692 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:00.194+0000 7f35a35c9700 1 -- 172.21.15.7:0/2721328961 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f359c10a250 con 0x7f359c105e60 2023-07-20T02:45:00.693 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:00.194+0000 7f35927fc700 1 -- 172.21.15.7:0/2721328961 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f3598018350 con 0x7f359c105e60 2023-07-20T02:45:00.693 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:00.197+0000 7f35a35c9700 1 -- 172.21.15.7:0/2721328961 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f3588038850 msgr2=0x7f358803ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:00.693 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:00.197+0000 7f35a35c9700 1 --2- 172.21.15.7:0/2721328961 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f3588038850 0x7f358803ad10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f358c006fd0 tx=0x7f358c006e40 comp rx=0 tx=0).stop 2023-07-20T02:45:00.693 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:00.197+0000 7f35a35c9700 1 -- 172.21.15.7:0/2721328961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f359c105e60 msgr2=0x7f359c19bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:00.693 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:00.197+0000 7f35a35c9700 1 --2- 172.21.15.7:0/2721328961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f359c105e60 0x7f359c19bd50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3598011980 tx=0x7f3598011a60 comp rx=0 tx=0).stop 2023-07-20T02:45:00.693 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:00.197+0000 7f35a35c9700 1 -- 172.21.15.7:0/2721328961 shutdown_connections 2023-07-20T02:45:00.693 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:00.197+0000 7f35a35c9700 1 --2- 172.21.15.7:0/2721328961 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f3588038850 0x7f358803ad10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:00.694 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:00.197+0000 7f35a35c9700 1 --2- 172.21.15.7:0/2721328961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f359c105e60 0x7f359c19bd50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:00.694 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:00.197+0000 7f35a35c9700 1 -- 172.21.15.7:0/2721328961 >> 172.21.15.7:0/2721328961 conn(0x7f359c101440 msgr2=0x7f359c102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:45:00.694 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:00.197+0000 7f35a35c9700 1 -- 172.21.15.7:0/2721328961 shutdown_connections 2023-07-20T02:45:00.694 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:00.197+0000 7f35a35c9700 1 -- 172.21.15.7:0/2721328961 wait complete. 2023-07-20T02:45:00.943 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:00 smithi007 ceph-mon[126353]: mgrmap e13: x(active, since 6s) 2023-07-20T02:45:02.063 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.210+0000 7f877e822700 1 Processor -- start 2023-07-20T02:45:02.063 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.210+0000 7f877e822700 1 -- start start 2023-07-20T02:45:02.063 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.211+0000 7f877e822700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f877807be90 0x7f877807a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:02.064 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.211+0000 7f877e822700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877807a8c0 con 0x7f877807be90 2023-07-20T02:45:02.064 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.211+0000 7f8777fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f877807be90 0x7f877807a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:02.064 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.211+0000 7f8777fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f877807be90 0x7f877807a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60608/0 (socket says 172.21.15.7:60608) 2023-07-20T02:45:02.064 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.211+0000 7f8777fff700 1 -- 172.21.15.7:0/1094018110 learned_addr learned my addr 172.21.15.7:0/1094018110 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:45:02.064 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.211+0000 7f8777fff700 1 -- 172.21.15.7:0/1094018110 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f877807aa00 con 0x7f877807be90 2023-07-20T02:45:02.065 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.212+0000 7f8777fff700 1 --2- 172.21.15.7:0/1094018110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f877807be90 0x7f877807a2f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f87680098a0 tx=0x7f8768009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da1e45050a14f2ab server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:45:02.065 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.212+0000 7f8776ffd700 1 -- 172.21.15.7:0/1094018110 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8768004160 con 0x7f877807be90 2023-07-20T02:45:02.065 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.212+0000 7f8776ffd700 1 -- 172.21.15.7:0/1094018110 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f87680042c0 con 0x7f877807be90 2023-07-20T02:45:02.065 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.212+0000 7f8776ffd700 1 -- 172.21.15.7:0/1094018110 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87680119b0 con 0x7f877807be90 2023-07-20T02:45:02.065 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.213+0000 7f877e822700 1 -- 172.21.15.7:0/1094018110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f877807be90 msgr2=0x7f877807a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:02.065 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.213+0000 7f877e822700 1 --2- 172.21.15.7:0/1094018110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f877807be90 0x7f877807a2f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f87680098a0 tx=0x7f8768009bb0 comp rx=0 tx=0).stop 2023-07-20T02:45:02.066 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.213+0000 7f877e822700 1 -- 172.21.15.7:0/1094018110 shutdown_connections 2023-07-20T02:45:02.066 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.213+0000 7f877e822700 1 --2- 172.21.15.7:0/1094018110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f877807be90 0x7f877807a2f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:02.066 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.213+0000 7f877e822700 1 -- 172.21.15.7:0/1094018110 >> 172.21.15.7:0/1094018110 conn(0x7f8778101440 msgr2=0x7f8778103860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:45:02.066 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.213+0000 7f877e822700 1 -- 172.21.15.7:0/1094018110 shutdown_connections 2023-07-20T02:45:02.070 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.213+0000 7f877e822700 1 -- 172.21.15.7:0/1094018110 wait complete. 2023-07-20T02:45:02.070 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.214+0000 7f877e822700 1 Processor -- start 2023-07-20T02:45:02.070 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.214+0000 7f877e822700 1 -- start start 2023-07-20T02:45:02.070 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.214+0000 7f877e822700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f877807be90 0x7f877819bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:02.071 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.214+0000 7f877e822700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877819c2e0 con 0x7f877807be90 2023-07-20T02:45:02.071 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.214+0000 7f8777fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f877807be90 0x7f877819bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:02.071 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.214+0000 7f8777fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f877807be90 0x7f877819bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60618/0 (socket says 172.21.15.7:60618) 2023-07-20T02:45:02.071 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.214+0000 7f8777fff700 1 -- 172.21.15.7:0/1311716137 learned_addr learned my addr 172.21.15.7:0/1311716137 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:45:02.071 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.215+0000 7f8777fff700 1 -- 172.21.15.7:0/1311716137 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8768009550 con 0x7f877807be90 2023-07-20T02:45:02.071 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.215+0000 7f8777fff700 1 --2- 172.21.15.7:0/1311716137 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f877807be90 0x7f877819bda0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8768000c00 tx=0x7f8768011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:45:02.072 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.215+0000 7f87757fa700 1 -- 172.21.15.7:0/1311716137 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87680039f0 con 0x7f877807be90 2023-07-20T02:45:02.072 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.215+0000 7f87757fa700 1 -- 172.21.15.7:0/1311716137 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8768003b50 con 0x7f877807be90 2023-07-20T02:45:02.072 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.215+0000 7f87757fa700 1 -- 172.21.15.7:0/1311716137 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f876801a780 con 0x7f877807be90 2023-07-20T02:45:02.072 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.215+0000 7f877e822700 1 -- 172.21.15.7:0/1311716137 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f877819c4e0 con 0x7f877807be90 2023-07-20T02:45:02.072 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.215+0000 7f877e822700 1 -- 172.21.15.7:0/1311716137 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f877819c900 con 0x7f877807be90 2023-07-20T02:45:02.072 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.217+0000 7f87757fa700 1 -- 172.21.15.7:0/1311716137 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f8768024440 con 0x7f877807be90 2023-07-20T02:45:02.072 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.217+0000 7f877e822700 1 -- 172.21.15.7:0/1311716137 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8778061da0 con 0x7f877807be90 2023-07-20T02:45:02.072 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.217+0000 7f87757fa700 1 --2- 172.21.15.7:0/1311716137 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f8760038850 0x7f876003ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:02.073 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.217+0000 7f87757fa700 1 -- 172.21.15.7:0/1311716137 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f876804c900 con 0x7f877807be90 2023-07-20T02:45:02.073 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.217+0000 7f87777fe700 1 --2- 172.21.15.7:0/1311716137 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f8760038850 0x7f876003ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:02.073 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.218+0000 7f87777fe700 1 --2- 172.21.15.7:0/1311716137 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f8760038850 0x7f876003ad10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f876c006fd0 tx=0x7f876c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:45:02.073 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.222+0000 7f87757fa700 1 -- 172.21.15.7:0/1311716137 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8768018d90 con 0x7f877807be90 2023-07-20T02:45:02.073 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.538+0000 7f877e822700 1 -- 172.21.15.7:0/1311716137 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f877819f6f0 con 0x7f877807be90 2023-07-20T02:45:02.073 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.541+0000 7f87757fa700 1 -- 172.21.15.7:0/1311716137 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7f8768018350 con 0x7f877807be90 2023-07-20T02:45:02.073 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.543+0000 7f877e822700 1 -- 172.21.15.7:0/1311716137 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f8760038850 msgr2=0x7f876003ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:02.074 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.543+0000 7f877e822700 1 --2- 172.21.15.7:0/1311716137 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f8760038850 0x7f876003ad10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f876c006fd0 tx=0x7f876c006e40 comp rx=0 tx=0).stop 2023-07-20T02:45:02.074 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.543+0000 7f877e822700 1 -- 172.21.15.7:0/1311716137 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f877807be90 msgr2=0x7f877819bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:02.074 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.543+0000 7f877e822700 1 --2- 172.21.15.7:0/1311716137 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f877807be90 0x7f877819bda0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8768000c00 tx=0x7f8768011a30 comp rx=0 tx=0).stop 2023-07-20T02:45:02.074 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.544+0000 7f877e822700 1 -- 172.21.15.7:0/1311716137 shutdown_connections 2023-07-20T02:45:02.074 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.544+0000 7f877e822700 1 --2- 172.21.15.7:0/1311716137 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f8760038850 0x7f876003ad10 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:02.074 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.544+0000 7f877e822700 1 --2- 172.21.15.7:0/1311716137 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f877807be90 0x7f877819bda0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:02.074 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.544+0000 7f877e822700 1 -- 172.21.15.7:0/1311716137 >> 172.21.15.7:0/1311716137 conn(0x7f8778101440 msgr2=0x7f8778102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:45:02.074 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.544+0000 7f877e822700 1 -- 172.21.15.7:0/1311716137 shutdown_connections 2023-07-20T02:45:02.075 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2023-07-20T02:45:01.544+0000 7f877e822700 1 -- 172.21.15.7:0/1311716137 wait complete. 2023-07-20T02:45:02.075 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-07-20T02:45:02.075 INFO:teuthology.orchestra.run.smithi007.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-07-20T02:45:02.075 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:45:02.075 INFO:teuthology.orchestra.run.smithi007.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-07-20T02:45:02.075 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:45:02.075 INFO:teuthology.orchestra.run.smithi007.stdout:Or, if you are only running a single cluster on this host: 2023-07-20T02:45:02.076 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:45:02.076 INFO:teuthology.orchestra.run.smithi007.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2023-07-20T02:45:02.076 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:45:02.076 INFO:teuthology.orchestra.run.smithi007.stdout:Please consider enabling telemetry to help improve Ceph: 2023-07-20T02:45:02.076 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:45:02.076 INFO:teuthology.orchestra.run.smithi007.stdout: ceph telemetry on 2023-07-20T02:45:02.076 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:45:02.077 INFO:teuthology.orchestra.run.smithi007.stdout:For more information see: 2023-07-20T02:45:02.077 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:45:02.077 INFO:teuthology.orchestra.run.smithi007.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2023-07-20T02:45:02.077 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:45:02.078 INFO:teuthology.orchestra.run.smithi007.stdout:Bootstrap complete. 2023-07-20T02:45:02.129 INFO:tasks.cephadm:Fetching config... 2023-07-20T02:45:02.130 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-07-20T02:45:02.130 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-07-20T02:45:02.173 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-07-20T02:45:02.174 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-07-20T02:45:02.174 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-07-20T02:45:02.260 INFO:tasks.cephadm:Fetching mon keyring... 2023-07-20T02:45:02.261 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-07-20T02:45:02.261 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.a/keyring of=/dev/stdout 2023-07-20T02:45:02.372 INFO:tasks.cephadm:Fetching pub ssh key... 2023-07-20T02:45:02.373 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-07-20T02:45:02.373 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-07-20T02:45:02.457 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-07-20T02:45:02.457 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdC1rUDSVXOGKnQ7A1kKZo/ocL/B66GlwTKoUkAIU/wvj3aM/qBkt0JgxclGnrHVK3Kwf1s/1xzKd39Zp16At6Ofc8k6U1NeCQREZQzrgZ/97dUM4wY14yKXNdOoFNMdxPGjiWzEQeU7kaZ1xAy3h+CsyQGMNp7PyW8yIcKUeAfZ5XM8vdYiivp5ccz/egR6xcNQjAXg0Is3ELq16eauR32cVINdLNu/qQ2FvbUJ1ckShx8IkeJZj82pvo5amWWbC3znkU+PhlkYMleu0O1UrkTwAIyCauqDbFokUZ3zvK1xt7/9uvMuKSnoXAidhiFdL4dd2k4OQYXdQ52uoloGjajplx6Zzat/Us1NeBo5nQR45Og5yaN9ZJnw/E13BC2JkcGI90hQBlXkfzpP6+lP4+Bn9hwfQaBIDalyNc10TA3moSRR58xLybWj4m0mZaY3EyXnYsPqGevY7xMlWJwj0VuW8re1ebV7W+VQQ4VcYDgg2X0jarwpqjupKt54bgnok= ceph-1b04643c-26a7-11ee-9b34-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-20T02:45:02.604 INFO:teuthology.orchestra.run.smithi007.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdC1rUDSVXOGKnQ7A1kKZo/ocL/B66GlwTKoUkAIU/wvj3aM/qBkt0JgxclGnrHVK3Kwf1s/1xzKd39Zp16At6Ofc8k6U1NeCQREZQzrgZ/97dUM4wY14yKXNdOoFNMdxPGjiWzEQeU7kaZ1xAy3h+CsyQGMNp7PyW8yIcKUeAfZ5XM8vdYiivp5ccz/egR6xcNQjAXg0Is3ELq16eauR32cVINdLNu/qQ2FvbUJ1ckShx8IkeJZj82pvo5amWWbC3znkU+PhlkYMleu0O1UrkTwAIyCauqDbFokUZ3zvK1xt7/9uvMuKSnoXAidhiFdL4dd2k4OQYXdQ52uoloGjajplx6Zzat/Us1NeBo5nQR45Og5yaN9ZJnw/E13BC2JkcGI90hQBlXkfzpP6+lP4+Bn9hwfQaBIDalyNc10TA3moSRR58xLybWj4m0mZaY3EyXnYsPqGevY7xMlWJwj0VuW8re1ebV7W+VQQ4VcYDgg2X0jarwpqjupKt54bgnok= ceph-1b04643c-26a7-11ee-9b34-001a4aab830c 2023-07-20T02:45:02.627 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:02 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/1311716137' entity='client.admin' 2023-07-20T02:45:02.637 DEBUG:teuthology.orchestra.run.smithi158:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdC1rUDSVXOGKnQ7A1kKZo/ocL/B66GlwTKoUkAIU/wvj3aM/qBkt0JgxclGnrHVK3Kwf1s/1xzKd39Zp16At6Ofc8k6U1NeCQREZQzrgZ/97dUM4wY14yKXNdOoFNMdxPGjiWzEQeU7kaZ1xAy3h+CsyQGMNp7PyW8yIcKUeAfZ5XM8vdYiivp5ccz/egR6xcNQjAXg0Is3ELq16eauR32cVINdLNu/qQ2FvbUJ1ckShx8IkeJZj82pvo5amWWbC3znkU+PhlkYMleu0O1UrkTwAIyCauqDbFokUZ3zvK1xt7/9uvMuKSnoXAidhiFdL4dd2k4OQYXdQ52uoloGjajplx6Zzat/Us1NeBo5nQR45Og5yaN9ZJnw/E13BC2JkcGI90hQBlXkfzpP6+lP4+Bn9hwfQaBIDalyNc10TA3moSRR58xLybWj4m0mZaY3EyXnYsPqGevY7xMlWJwj0VuW8re1ebV7W+VQQ4VcYDgg2X0jarwpqjupKt54bgnok= ceph-1b04643c-26a7-11ee-9b34-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-20T02:45:02.684 INFO:teuthology.orchestra.run.smithi158.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdC1rUDSVXOGKnQ7A1kKZo/ocL/B66GlwTKoUkAIU/wvj3aM/qBkt0JgxclGnrHVK3Kwf1s/1xzKd39Zp16At6Ofc8k6U1NeCQREZQzrgZ/97dUM4wY14yKXNdOoFNMdxPGjiWzEQeU7kaZ1xAy3h+CsyQGMNp7PyW8yIcKUeAfZ5XM8vdYiivp5ccz/egR6xcNQjAXg0Is3ELq16eauR32cVINdLNu/qQ2FvbUJ1ckShx8IkeJZj82pvo5amWWbC3znkU+PhlkYMleu0O1UrkTwAIyCauqDbFokUZ3zvK1xt7/9uvMuKSnoXAidhiFdL4dd2k4OQYXdQ52uoloGjajplx6Zzat/Us1NeBo5nQR45Og5yaN9ZJnw/E13BC2JkcGI90hQBlXkfzpP6+lP4+Bn9hwfQaBIDalyNc10TA3moSRR58xLybWj4m0mZaY3EyXnYsPqGevY7xMlWJwj0VuW8re1ebV7W+VQQ4VcYDgg2X0jarwpqjupKt54bgnok= ceph-1b04643c-26a7-11ee-9b34-001a4aab830c 2023-07-20T02:45:02.697 DEBUG:teuthology.orchestra.run.smithi188:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdC1rUDSVXOGKnQ7A1kKZo/ocL/B66GlwTKoUkAIU/wvj3aM/qBkt0JgxclGnrHVK3Kwf1s/1xzKd39Zp16At6Ofc8k6U1NeCQREZQzrgZ/97dUM4wY14yKXNdOoFNMdxPGjiWzEQeU7kaZ1xAy3h+CsyQGMNp7PyW8yIcKUeAfZ5XM8vdYiivp5ccz/egR6xcNQjAXg0Is3ELq16eauR32cVINdLNu/qQ2FvbUJ1ckShx8IkeJZj82pvo5amWWbC3znkU+PhlkYMleu0O1UrkTwAIyCauqDbFokUZ3zvK1xt7/9uvMuKSnoXAidhiFdL4dd2k4OQYXdQ52uoloGjajplx6Zzat/Us1NeBo5nQR45Og5yaN9ZJnw/E13BC2JkcGI90hQBlXkfzpP6+lP4+Bn9hwfQaBIDalyNc10TA3moSRR58xLybWj4m0mZaY3EyXnYsPqGevY7xMlWJwj0VuW8re1ebV7W+VQQ4VcYDgg2X0jarwpqjupKt54bgnok= ceph-1b04643c-26a7-11ee-9b34-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-20T02:45:02.742 INFO:teuthology.orchestra.run.smithi188.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdC1rUDSVXOGKnQ7A1kKZo/ocL/B66GlwTKoUkAIU/wvj3aM/qBkt0JgxclGnrHVK3Kwf1s/1xzKd39Zp16At6Ofc8k6U1NeCQREZQzrgZ/97dUM4wY14yKXNdOoFNMdxPGjiWzEQeU7kaZ1xAy3h+CsyQGMNp7PyW8yIcKUeAfZ5XM8vdYiivp5ccz/egR6xcNQjAXg0Is3ELq16eauR32cVINdLNu/qQ2FvbUJ1ckShx8IkeJZj82pvo5amWWbC3znkU+PhlkYMleu0O1UrkTwAIyCauqDbFokUZ3zvK1xt7/9uvMuKSnoXAidhiFdL4dd2k4OQYXdQ52uoloGjajplx6Zzat/Us1NeBo5nQR45Og5yaN9ZJnw/E13BC2JkcGI90hQBlXkfzpP6+lP4+Bn9hwfQaBIDalyNc10TA3moSRR58xLybWj4m0mZaY3EyXnYsPqGevY7xMlWJwj0VuW8re1ebV7W+VQQ4VcYDgg2X0jarwpqjupKt54bgnok= ceph-1b04643c-26a7-11ee-9b34-001a4aab830c 2023-07-20T02:45:02.755 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-07-20T02:45:03.692 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:45:05.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:04 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:05.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:04 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:05.076 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:04 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi007", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:45:05.076 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:04 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:05.076 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:04 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:45:05.076 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:04 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:05.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.439+0000 7f7d26cec700 1 -- 172.21.15.7:0/4176651237 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d201001e0 msgr2=0x7f7d20100600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:05.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.439+0000 7f7d26cec700 1 --2- 172.21.15.7:0/4176651237 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d201001e0 0x7f7d20100600 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7d14009910 tx=0x7f7d14009c20 comp rx=0 tx=0).stop 2023-07-20T02:45:05.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.439+0000 7f7d26cec700 1 -- 172.21.15.7:0/4176651237 shutdown_connections 2023-07-20T02:45:05.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.439+0000 7f7d26cec700 1 --2- 172.21.15.7:0/4176651237 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d201001e0 0x7f7d20100600 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:05.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.439+0000 7f7d26cec700 1 -- 172.21.15.7:0/4176651237 >> 172.21.15.7:0/4176651237 conn(0x7f7d200fb7a0 msgr2=0x7f7d200fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:45:05.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.439+0000 7f7d26cec700 1 -- 172.21.15.7:0/4176651237 shutdown_connections 2023-07-20T02:45:05.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.439+0000 7f7d26cec700 1 -- 172.21.15.7:0/4176651237 wait complete. 2023-07-20T02:45:05.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.440+0000 7f7d26cec700 1 Processor -- start 2023-07-20T02:45:05.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.441+0000 7f7d26cec700 1 -- start start 2023-07-20T02:45:05.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.441+0000 7f7d26cec700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d201001e0 0x7f7d201974c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:05.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.441+0000 7f7d26cec700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d20197a00 con 0x7f7d201001e0 2023-07-20T02:45:05.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.441+0000 7f7d24a88700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d201001e0 0x7f7d201974c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:05.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.441+0000 7f7d24a88700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d201001e0 0x7f7d201974c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52984/0 (socket says 172.21.15.7:52984) 2023-07-20T02:45:05.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.441+0000 7f7d24a88700 1 -- 172.21.15.7:0/3307090307 learned_addr learned my addr 172.21.15.7:0/3307090307 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:45:05.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.441+0000 7f7d24a88700 1 -- 172.21.15.7:0/3307090307 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d140095f0 con 0x7f7d201001e0 2023-07-20T02:45:05.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.442+0000 7f7d24a88700 1 --2- 172.21.15.7:0/3307090307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d201001e0 0x7f7d201974c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7d14004f40 tx=0x7f7d140057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:45:05.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.442+0000 7f7d1dffb700 1 -- 172.21.15.7:0/3307090307 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7d1401d070 con 0x7f7d201001e0 2023-07-20T02:45:05.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.442+0000 7f7d26cec700 1 -- 172.21.15.7:0/3307090307 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d20197c00 con 0x7f7d201001e0 2023-07-20T02:45:05.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.442+0000 7f7d1dffb700 1 -- 172.21.15.7:0/3307090307 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7d1400bc10 con 0x7f7d201001e0 2023-07-20T02:45:05.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.442+0000 7f7d1dffb700 1 -- 172.21.15.7:0/3307090307 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7d14003a70 con 0x7f7d201001e0 2023-07-20T02:45:05.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.442+0000 7f7d26cec700 1 -- 172.21.15.7:0/3307090307 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d201980a0 con 0x7f7d201001e0 2023-07-20T02:45:05.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.443+0000 7f7d1dffb700 1 -- 172.21.15.7:0/3307090307 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f7d14003ce0 con 0x7f7d201001e0 2023-07-20T02:45:05.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.443+0000 7f7d26cec700 1 -- 172.21.15.7:0/3307090307 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d201910f0 con 0x7f7d201001e0 2023-07-20T02:45:05.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.444+0000 7f7d1dffb700 1 --2- 172.21.15.7:0/3307090307 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f7d10038800 0x7f7d1003acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:05.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.444+0000 7f7d1dffb700 1 -- 172.21.15.7:0/3307090307 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7d1404c530 con 0x7f7d201001e0 2023-07-20T02:45:05.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.444+0000 7f7d1ffff700 1 --2- 172.21.15.7:0/3307090307 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f7d10038800 0x7f7d1003acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:05.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.445+0000 7f7d1ffff700 1 --2- 172.21.15.7:0/3307090307 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f7d10038800 0x7f7d1003acc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c006fd0 tx=0x7f7d0c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:45:05.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.449+0000 7f7d1dffb700 1 -- 172.21.15.7:0/3307090307 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7d1402aba0 con 0x7f7d201001e0 2023-07-20T02:45:05.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.661+0000 7f7d26cec700 1 -- 172.21.15.7:0/3307090307 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f7d2002c970 con 0x7f7d201001e0 2023-07-20T02:45:05.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.666+0000 7f7d1dffb700 1 -- 172.21.15.7:0/3307090307 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f7d14026070 con 0x7f7d201001e0 2023-07-20T02:45:05.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.669+0000 7f7d26cec700 1 -- 172.21.15.7:0/3307090307 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f7d10038800 msgr2=0x7f7d1003acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:05.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.669+0000 7f7d26cec700 1 --2- 172.21.15.7:0/3307090307 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f7d10038800 0x7f7d1003acc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c006fd0 tx=0x7f7d0c006e40 comp rx=0 tx=0).stop 2023-07-20T02:45:05.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.669+0000 7f7d26cec700 1 -- 172.21.15.7:0/3307090307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d201001e0 msgr2=0x7f7d201974c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:05.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.669+0000 7f7d26cec700 1 --2- 172.21.15.7:0/3307090307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d201001e0 0x7f7d201974c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7d14004f40 tx=0x7f7d140057c0 comp rx=0 tx=0).stop 2023-07-20T02:45:05.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.669+0000 7f7d26cec700 1 -- 172.21.15.7:0/3307090307 shutdown_connections 2023-07-20T02:45:05.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.669+0000 7f7d26cec700 1 --2- 172.21.15.7:0/3307090307 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f7d10038800 0x7f7d1003acc0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:05.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.669+0000 7f7d26cec700 1 --2- 172.21.15.7:0/3307090307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d201001e0 0x7f7d201974c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:05.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.669+0000 7f7d26cec700 1 -- 172.21.15.7:0/3307090307 >> 172.21.15.7:0/3307090307 conn(0x7f7d200fb7a0 msgr2=0x7f7d200fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:45:05.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.670+0000 7f7d26cec700 1 -- 172.21.15.7:0/3307090307 shutdown_connections 2023-07-20T02:45:05.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:05.670+0000 7f7d26cec700 1 -- 172.21.15.7:0/3307090307 wait complete. 2023-07-20T02:45:06.176 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-07-20T02:45:06.177 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-07-20T02:45:06.907 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:45:06.943 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:06 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/3307090307' entity='client.admin' 2023-07-20T02:45:06.943 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:06 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:45:06.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:06 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:08.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.421+0000 7fe405f3b700 1 -- 172.21.15.7:0/3186026785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe400100280 msgr2=0x7fe4001006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:08.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.421+0000 7fe405f3b700 1 --2- 172.21.15.7:0/3186026785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe400100280 0x7fe4001006a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe3f0009910 tx=0x7fe3f0009c20 comp rx=0 tx=0).stop 2023-07-20T02:45:08.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.421+0000 7fe405f3b700 1 -- 172.21.15.7:0/3186026785 shutdown_connections 2023-07-20T02:45:08.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.421+0000 7fe405f3b700 1 --2- 172.21.15.7:0/3186026785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe400100280 0x7fe4001006a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:08.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.421+0000 7fe405f3b700 1 -- 172.21.15.7:0/3186026785 >> 172.21.15.7:0/3186026785 conn(0x7fe4000fb800 msgr2=0x7fe4000fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:45:08.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.421+0000 7fe405f3b700 1 -- 172.21.15.7:0/3186026785 shutdown_connections 2023-07-20T02:45:08.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.421+0000 7fe405f3b700 1 -- 172.21.15.7:0/3186026785 wait complete. 2023-07-20T02:45:08.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.422+0000 7fe405f3b700 1 Processor -- start 2023-07-20T02:45:08.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.422+0000 7fe405f3b700 1 -- start start 2023-07-20T02:45:08.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.423+0000 7fe405f3b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe400100280 0x7fe400195390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:08.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.423+0000 7fe405f3b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4001958d0 con 0x7fe400100280 2023-07-20T02:45:08.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.423+0000 7fe3ff7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe400100280 0x7fe400195390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:08.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.423+0000 7fe3ff7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe400100280 0x7fe400195390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53014/0 (socket says 172.21.15.7:53014) 2023-07-20T02:45:08.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.423+0000 7fe3ff7fe700 1 -- 172.21.15.7:0/1816508388 learned_addr learned my addr 172.21.15.7:0/1816508388 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:45:08.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.423+0000 7fe3ff7fe700 1 -- 172.21.15.7:0/1816508388 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f00095f0 con 0x7fe400100280 2023-07-20T02:45:08.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.423+0000 7fe3ff7fe700 1 --2- 172.21.15.7:0/1816508388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe400100280 0x7fe400195390 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe3f0004f40 tx=0x7fe3f00057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:45:08.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.424+0000 7fe3fd7fa700 1 -- 172.21.15.7:0/1816508388 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe3f001d070 con 0x7fe400100280 2023-07-20T02:45:08.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.424+0000 7fe3fd7fa700 1 -- 172.21.15.7:0/1816508388 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe3f000bc10 con 0x7fe400100280 2023-07-20T02:45:08.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.424+0000 7fe3fd7fa700 1 -- 172.21.15.7:0/1816508388 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe3f0003a70 con 0x7fe400100280 2023-07-20T02:45:08.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.424+0000 7fe405f3b700 1 -- 172.21.15.7:0/1816508388 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe400195ad0 con 0x7fe400100280 2023-07-20T02:45:08.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.424+0000 7fe405f3b700 1 -- 172.21.15.7:0/1816508388 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe400195f70 con 0x7fe400100280 2023-07-20T02:45:08.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.425+0000 7fe405f3b700 1 -- 172.21.15.7:0/1816508388 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe40018ed90 con 0x7fe400100280 2023-07-20T02:45:08.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.425+0000 7fe3fd7fa700 1 -- 172.21.15.7:0/1816508388 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7fe3f0003ce0 con 0x7fe400100280 2023-07-20T02:45:08.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.426+0000 7fe3fd7fa700 1 --2- 172.21.15.7:0/1816508388 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fe3ec038800 0x7fe3ec03acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:08.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.426+0000 7fe3fd7fa700 1 -- 172.21.15.7:0/1816508388 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe3f004c4f0 con 0x7fe400100280 2023-07-20T02:45:08.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.426+0000 7fe3f7fff700 1 --2- 172.21.15.7:0/1816508388 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fe3ec038800 0x7fe3ec03acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:08.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.427+0000 7fe3f7fff700 1 --2- 172.21.15.7:0/1816508388 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fe3ec038800 0x7fe3ec03acc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8006fd0 tx=0x7fe3e8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:45:08.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.431+0000 7fe3fd7fa700 1 -- 172.21.15.7:0/1816508388 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe3f002aba0 con 0x7fe400100280 2023-07-20T02:45:08.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.640+0000 7fe405f3b700 1 -- 172.21.15.7:0/1816508388 --> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fe400060c40 con 0x7fe3ec038800 2023-07-20T02:45:08.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.643+0000 7fe3fd7fa700 1 -- 172.21.15.7:0/1816508388 <== mgr.14152 v2:172.21.15.7:6800/1354542137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe400060c40 con 0x7fe3ec038800 2023-07-20T02:45:08.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.646+0000 7fe405f3b700 1 -- 172.21.15.7:0/1816508388 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fe3ec038800 msgr2=0x7fe3ec03acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:08.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.646+0000 7fe405f3b700 1 --2- 172.21.15.7:0/1816508388 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fe3ec038800 0x7fe3ec03acc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8006fd0 tx=0x7fe3e8006e40 comp rx=0 tx=0).stop 2023-07-20T02:45:08.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.646+0000 7fe405f3b700 1 -- 172.21.15.7:0/1816508388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe400100280 msgr2=0x7fe400195390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:08.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.646+0000 7fe405f3b700 1 --2- 172.21.15.7:0/1816508388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe400100280 0x7fe400195390 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe3f0004f40 tx=0x7fe3f00057c0 comp rx=0 tx=0).stop 2023-07-20T02:45:08.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.646+0000 7fe405f3b700 1 -- 172.21.15.7:0/1816508388 shutdown_connections 2023-07-20T02:45:08.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.646+0000 7fe405f3b700 1 --2- 172.21.15.7:0/1816508388 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fe3ec038800 0x7fe3ec03acc0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:08.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.646+0000 7fe405f3b700 1 --2- 172.21.15.7:0/1816508388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe400100280 0x7fe400195390 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:08.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.646+0000 7fe405f3b700 1 -- 172.21.15.7:0/1816508388 >> 172.21.15.7:0/1816508388 conn(0x7fe4000fb800 msgr2=0x7fe4000fc5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:45:08.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.646+0000 7fe405f3b700 1 -- 172.21.15.7:0/1816508388 shutdown_connections 2023-07-20T02:45:08.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:08.646+0000 7fe405f3b700 1 -- 172.21.15.7:0/1816508388 wait complete. 2023-07-20T02:45:09.188 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi158 2023-07-20T02:45:09.189 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-07-20T02:45:09.189 DEBUG:teuthology.orchestra.run.smithi158:> dd of=/etc/ceph/ceph.conf 2023-07-20T02:45:09.209 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-07-20T02:45:09.210 DEBUG:teuthology.orchestra.run.smithi158:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T02:45:09.274 INFO:tasks.cephadm:Adding host smithi158 to orchestrator... 2023-07-20T02:45:09.274 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph orch host add smithi158 2023-07-20T02:45:09.943 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:09 smithi007 ceph-mon[126353]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:45:09.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:09 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:09.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:09 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:45:09.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:09 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:45:09.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:09 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:45:09.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:09 smithi007 ceph-mon[126353]: Updating smithi007:/etc/ceph/ceph.conf 2023-07-20T02:45:09.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:09 smithi007 ceph-mon[126353]: Updating smithi007:/var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T02:45:10.019 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:45:10.864 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:10 smithi007 ceph-mon[126353]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2023-07-20T02:45:10.864 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:10 smithi007 ceph-mon[126353]: Updating smithi007:/var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T02:45:11.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:11.508+0000 7f085b28c700 1 -- 172.21.15.7:0/75642865 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0854100460 msgr2=0x7f0854100840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:11.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:11.509+0000 7f085b28c700 1 --2- 172.21.15.7:0/75642865 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0854100460 0x7f0854100840 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0850009910 tx=0x7f0850009c20 comp rx=0 tx=0).stop 2023-07-20T02:45:11.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:11.509+0000 7f085b28c700 1 -- 172.21.15.7:0/75642865 shutdown_connections 2023-07-20T02:45:11.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:11.509+0000 7f085b28c700 1 --2- 172.21.15.7:0/75642865 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0854100460 0x7f0854100840 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:11.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:11.509+0000 7f085b28c700 1 -- 172.21.15.7:0/75642865 >> 172.21.15.7:0/75642865 conn(0x7f08540fbd30 msgr2=0x7f08540fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:45:11.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:11.509+0000 7f085b28c700 1 -- 172.21.15.7:0/75642865 shutdown_connections 2023-07-20T02:45:11.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:11.509+0000 7f085b28c700 1 -- 172.21.15.7:0/75642865 wait complete. 2023-07-20T02:45:11.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:11.510+0000 7f085b28c700 1 Processor -- start 2023-07-20T02:45:11.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:11.510+0000 7f085b28c700 1 -- start start 2023-07-20T02:45:11.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:11.511+0000 7f085b28c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0854100460 0x7f0854195740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:11.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:11.511+0000 7f085b28c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0854195c80 con 0x7f0854100460 2023-07-20T02:45:11.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:11.511+0000 7f0859028700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0854100460 0x7f0854195740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:11.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:11.511+0000 7f0859028700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0854100460 0x7f0854195740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53034/0 (socket says 172.21.15.7:53034) 2023-07-20T02:45:11.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:11.511+0000 7f0859028700 1 -- 172.21.15.7:0/3333228520 learned_addr learned my addr 172.21.15.7:0/3333228520 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:45:11.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:11.512+0000 7f0859028700 1 -- 172.21.15.7:0/3333228520 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08500095f0 con 0x7f0854100460 2023-07-20T02:45:11.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:11.512+0000 7f0859028700 1 --2- 172.21.15.7:0/3333228520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0854100460 0x7f0854195740 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0850004f40 tx=0x7f0850005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:45:11.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:11.512+0000 7f084a7fc700 1 -- 172.21.15.7:0/3333228520 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f085001d070 con 0x7f0854100460 2023-07-20T02:45:11.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:11.512+0000 7f084a7fc700 1 -- 172.21.15.7:0/3333228520 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0850004370 con 0x7f0854100460 2023-07-20T02:45:11.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:11.512+0000 7f085b28c700 1 -- 172.21.15.7:0/3333228520 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0854195e80 con 0x7f0854100460 2023-07-20T02:45:11.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:11.512+0000 7f084a7fc700 1 -- 172.21.15.7:0/3333228520 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f08500039b0 con 0x7f0854100460 2023-07-20T02:45:11.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:11.512+0000 7f085b28c700 1 -- 172.21.15.7:0/3333228520 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08541962a0 con 0x7f0854100460 2023-07-20T02:45:11.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:11.513+0000 7f084a7fc700 1 -- 172.21.15.7:0/3333228520 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f0850003c70 con 0x7f0854100460 2023-07-20T02:45:11.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:11.513+0000 7f085b28c700 1 -- 172.21.15.7:0/3333228520 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0854061da0 con 0x7f0854100460 2023-07-20T02:45:11.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:11.514+0000 7f084a7fc700 1 --2- 172.21.15.7:0/3333228520 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f08400387b0 0x7f084003ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:11.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:11.514+0000 7f084a7fc700 1 -- 172.21.15.7:0/3333228520 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f085004c5e0 con 0x7f0854100460 2023-07-20T02:45:11.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:11.514+0000 7f0858827700 1 --2- 172.21.15.7:0/3333228520 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f08400387b0 0x7f084003ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:11.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:11.515+0000 7f0858827700 1 --2- 172.21.15.7:0/3333228520 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f08400387b0 0x7f084003ac70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0844006fd0 tx=0x7f0844006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:45:11.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:11.519+0000 7f084a7fc700 1 -- 172.21.15.7:0/3333228520 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f085002a980 con 0x7f0854100460 2023-07-20T02:45:11.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:11.729+0000 7f085b28c700 1 -- 172.21.15.7:0/3333228520 --> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi158", "target": ["mon-mgr", ""]}) v1 -- 0x7f08541076e0 con 0x7f08400387b0 2023-07-20T02:45:12.193 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:11 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:12.193 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:11 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:12.193 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:11 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:13.193 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:12 smithi007 ceph-mon[126353]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi158", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:45:13.193 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:12 smithi007 ceph-mon[126353]: Deploying cephadm binary to smithi158 2023-07-20T02:45:14.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:14.139+0000 7f084a7fc700 1 -- 172.21.15.7:0/3333228520 <== mgr.14152 v2:172.21.15.7:6800/1354542137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f08541076e0 con 0x7f08400387b0 2023-07-20T02:45:14.148 INFO:teuthology.orchestra.run.smithi007.stdout:Added host 'smithi158' with addr '172.21.15.158' 2023-07-20T02:45:14.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:14.142+0000 7f085b28c700 1 -- 172.21.15.7:0/3333228520 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f08400387b0 msgr2=0x7f084003ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:14.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:14.142+0000 7f085b28c700 1 --2- 172.21.15.7:0/3333228520 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f08400387b0 0x7f084003ac70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0844006fd0 tx=0x7f0844006e40 comp rx=0 tx=0).stop 2023-07-20T02:45:14.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:14.142+0000 7f085b28c700 1 -- 172.21.15.7:0/3333228520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0854100460 msgr2=0x7f0854195740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:14.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:14.142+0000 7f085b28c700 1 --2- 172.21.15.7:0/3333228520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0854100460 0x7f0854195740 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0850004f40 tx=0x7f0850005020 comp rx=0 tx=0).stop 2023-07-20T02:45:14.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:14.142+0000 7f085b28c700 1 -- 172.21.15.7:0/3333228520 shutdown_connections 2023-07-20T02:45:14.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:14.142+0000 7f085b28c700 1 --2- 172.21.15.7:0/3333228520 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f08400387b0 0x7f084003ac70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:14.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:14.142+0000 7f085b28c700 1 --2- 172.21.15.7:0/3333228520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0854100460 0x7f0854195740 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:14.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:14.142+0000 7f085b28c700 1 -- 172.21.15.7:0/3333228520 >> 172.21.15.7:0/3333228520 conn(0x7f08540fbd30 msgr2=0x7f0854105fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:45:14.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:14.143+0000 7f085b28c700 1 -- 172.21.15.7:0/3333228520 shutdown_connections 2023-07-20T02:45:14.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:14.143+0000 7f085b28c700 1 -- 172.21.15.7:0/3333228520 wait complete. 2023-07-20T02:45:14.707 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph orch host ls --format=json 2023-07-20T02:45:14.770 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:14 smithi007 ceph-mon[126353]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:14.770 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:14 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:14.771 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:14 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:45:15.333 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:45:15.885 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:15 smithi007 ceph-mon[126353]: Added host smithi158 2023-07-20T02:45:15.885 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:15 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:15.885 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:15 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:16.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:16.849+0000 7feee55e4700 1 -- 172.21.15.7:0/1902192711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feee0106290 msgr2=0x7feee0106670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:16.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:16.849+0000 7feee55e4700 1 --2- 172.21.15.7:0/1902192711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feee0106290 0x7feee0106670 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7feed0009910 tx=0x7feed0009c20 comp rx=0 tx=0).stop 2023-07-20T02:45:16.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:16.849+0000 7feee55e4700 1 -- 172.21.15.7:0/1902192711 shutdown_connections 2023-07-20T02:45:16.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:16.849+0000 7feee55e4700 1 --2- 172.21.15.7:0/1902192711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feee0106290 0x7feee0106670 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:16.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:16.849+0000 7feee55e4700 1 -- 172.21.15.7:0/1902192711 >> 172.21.15.7:0/1902192711 conn(0x7feee0074c70 msgr2=0x7feee0075080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:45:16.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:16.849+0000 7feee55e4700 1 -- 172.21.15.7:0/1902192711 shutdown_connections 2023-07-20T02:45:16.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:16.849+0000 7feee55e4700 1 -- 172.21.15.7:0/1902192711 wait complete. 2023-07-20T02:45:16.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:16.850+0000 7feee55e4700 1 Processor -- start 2023-07-20T02:45:16.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:16.850+0000 7feee55e4700 1 -- start start 2023-07-20T02:45:16.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:16.851+0000 7feee55e4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feee0106290 0x7feee0195690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:16.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:16.851+0000 7feee55e4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee0195bd0 con 0x7feee0106290 2023-07-20T02:45:16.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:16.851+0000 7feedeffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feee0106290 0x7feee0195690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:16.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:16.851+0000 7feedeffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feee0106290 0x7feee0195690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36406/0 (socket says 172.21.15.7:36406) 2023-07-20T02:45:16.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:16.851+0000 7feedeffd700 1 -- 172.21.15.7:0/1586060114 learned_addr learned my addr 172.21.15.7:0/1586060114 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:45:16.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:16.851+0000 7feedeffd700 1 -- 172.21.15.7:0/1586060114 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed00095f0 con 0x7feee0106290 2023-07-20T02:45:16.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:16.852+0000 7feedeffd700 1 --2- 172.21.15.7:0/1586060114 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feee0106290 0x7feee0195690 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7feed0004f40 tx=0x7feed0005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:45:16.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:16.852+0000 7feec7fff700 1 -- 172.21.15.7:0/1586060114 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feed001d070 con 0x7feee0106290 2023-07-20T02:45:16.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:16.852+0000 7feec7fff700 1 -- 172.21.15.7:0/1586060114 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feed0004370 con 0x7feee0106290 2023-07-20T02:45:16.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:16.852+0000 7feec7fff700 1 -- 172.21.15.7:0/1586060114 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feed0003970 con 0x7feee0106290 2023-07-20T02:45:16.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:16.852+0000 7feee55e4700 1 -- 172.21.15.7:0/1586060114 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee0195e30 con 0x7feee0106290 2023-07-20T02:45:16.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:16.852+0000 7feee55e4700 1 -- 172.21.15.7:0/1586060114 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feee01962b0 con 0x7feee0106290 2023-07-20T02:45:16.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:16.853+0000 7feee55e4700 1 -- 172.21.15.7:0/1586060114 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee0061da0 con 0x7feee0106290 2023-07-20T02:45:16.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:16.853+0000 7feec7fff700 1 -- 172.21.15.7:0/1586060114 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7feed0003be0 con 0x7feee0106290 2023-07-20T02:45:16.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:16.854+0000 7feec7fff700 1 --2- 172.21.15.7:0/1586060114 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7feec8038800 0x7feec803acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:16.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:16.854+0000 7feec7fff700 1 -- 172.21.15.7:0/1586060114 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7feed004c560 con 0x7feee0106290 2023-07-20T02:45:16.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:16.854+0000 7feede7fc700 1 --2- 172.21.15.7:0/1586060114 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7feec8038800 0x7feec803acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:16.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:16.854+0000 7feede7fc700 1 --2- 172.21.15.7:0/1586060114 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7feec8038800 0x7feec803acc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7feed4006fd0 tx=0x7feed4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:45:16.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:16.859+0000 7feec7fff700 1 -- 172.21.15.7:0/1586060114 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7feed002a590 con 0x7feee0106290 2023-07-20T02:45:16.943 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:16 smithi007 ceph-mon[126353]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:17.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:17.071+0000 7feee55e4700 1 -- 172.21.15.7:0/1586060114 --> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7feee01050d0 con 0x7feec8038800 2023-07-20T02:45:17.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:17.072+0000 7feec7fff700 1 -- 172.21.15.7:0/1586060114 <== mgr.14152 v2:172.21.15.7:6800/1354542137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+159 (secure 0 0 0) 0x7feee01050d0 con 0x7feec8038800 2023-07-20T02:45:17.081 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:45:17.081 INFO:teuthology.orchestra.run.smithi007.stdout:[{"addr": "172.21.15.7", "hostname": "smithi007", "labels": [], "status": ""}, {"addr": "172.21.15.158", "hostname": "smithi158", "labels": [], "status": ""}] 2023-07-20T02:45:17.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:17.074+0000 7feee55e4700 1 -- 172.21.15.7:0/1586060114 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7feec8038800 msgr2=0x7feec803acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:17.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:17.074+0000 7feee55e4700 1 --2- 172.21.15.7:0/1586060114 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7feec8038800 0x7feec803acc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7feed4006fd0 tx=0x7feed4006e40 comp rx=0 tx=0).stop 2023-07-20T02:45:17.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:17.074+0000 7feee55e4700 1 -- 172.21.15.7:0/1586060114 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feee0106290 msgr2=0x7feee0195690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:17.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:17.074+0000 7feee55e4700 1 --2- 172.21.15.7:0/1586060114 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feee0106290 0x7feee0195690 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7feed0004f40 tx=0x7feed0005020 comp rx=0 tx=0).stop 2023-07-20T02:45:17.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:17.074+0000 7feee55e4700 1 -- 172.21.15.7:0/1586060114 shutdown_connections 2023-07-20T02:45:17.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:17.074+0000 7feee55e4700 1 --2- 172.21.15.7:0/1586060114 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7feec8038800 0x7feec803acc0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:17.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:17.074+0000 7feee55e4700 1 --2- 172.21.15.7:0/1586060114 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feee0106290 0x7feee0195690 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:17.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:17.074+0000 7feee55e4700 1 -- 172.21.15.7:0/1586060114 >> 172.21.15.7:0/1586060114 conn(0x7feee0074c70 msgr2=0x7feee00fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:45:17.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:17.075+0000 7feee55e4700 1 -- 172.21.15.7:0/1586060114 shutdown_connections 2023-07-20T02:45:17.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:17.075+0000 7feee55e4700 1 -- 172.21.15.7:0/1586060114 wait complete. 2023-07-20T02:45:17.621 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi188 2023-07-20T02:45:17.622 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2023-07-20T02:45:17.622 DEBUG:teuthology.orchestra.run.smithi188:> dd of=/etc/ceph/ceph.conf 2023-07-20T02:45:17.645 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2023-07-20T02:45:17.646 DEBUG:teuthology.orchestra.run.smithi188:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T02:45:17.707 INFO:tasks.cephadm:Adding host smithi188 to orchestrator... 2023-07-20T02:45:17.707 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph orch host add smithi188 2023-07-20T02:45:17.770 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:17 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:18.435 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:45:18.943 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:18 smithi007 ceph-mon[126353]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-20T02:45:18.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:18 smithi007 ceph-mon[126353]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:20.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:19.991+0000 7f279031c700 1 -- 172.21.15.7:0/2674927060 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2788100460 msgr2=0x7f2788100840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:20.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:19.991+0000 7f279031c700 1 --2- 172.21.15.7:0/2674927060 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2788100460 0x7f2788100840 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2784009910 tx=0x7f2784009c20 comp rx=0 tx=0).stop 2023-07-20T02:45:20.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:19.991+0000 7f279031c700 1 -- 172.21.15.7:0/2674927060 shutdown_connections 2023-07-20T02:45:20.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:19.991+0000 7f279031c700 1 --2- 172.21.15.7:0/2674927060 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2788100460 0x7f2788100840 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:20.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:19.991+0000 7f279031c700 1 -- 172.21.15.7:0/2674927060 >> 172.21.15.7:0/2674927060 conn(0x7f27880fbd30 msgr2=0x7f27880fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:45:20.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:19.991+0000 7f279031c700 1 -- 172.21.15.7:0/2674927060 shutdown_connections 2023-07-20T02:45:20.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:19.991+0000 7f279031c700 1 -- 172.21.15.7:0/2674927060 wait complete. 2023-07-20T02:45:20.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:19.992+0000 7f279031c700 1 Processor -- start 2023-07-20T02:45:20.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:19.992+0000 7f279031c700 1 -- start start 2023-07-20T02:45:20.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:19.993+0000 7f279031c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2788100460 0x7f2788072a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:20.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:19.993+0000 7f279031c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2788074950 con 0x7f2788100460 2023-07-20T02:45:20.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:19.993+0000 7f278e0b8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2788100460 0x7f2788072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:20.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:19.993+0000 7f278e0b8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2788100460 0x7f2788072a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36426/0 (socket says 172.21.15.7:36426) 2023-07-20T02:45:20.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:19.993+0000 7f278e0b8700 1 -- 172.21.15.7:0/947122905 learned_addr learned my addr 172.21.15.7:0/947122905 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:45:20.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:19.994+0000 7f278e0b8700 1 -- 172.21.15.7:0/947122905 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27840095f0 con 0x7f2788100460 2023-07-20T02:45:20.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:19.994+0000 7f278e0b8700 1 --2- 172.21.15.7:0/947122905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2788100460 0x7f2788072a70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2784004f40 tx=0x7f2784005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:45:20.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:19.994+0000 7f277f7fe700 1 -- 172.21.15.7:0/947122905 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f278401d070 con 0x7f2788100460 2023-07-20T02:45:20.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:19.994+0000 7f277f7fe700 1 -- 172.21.15.7:0/947122905 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2784004370 con 0x7f2788100460 2023-07-20T02:45:20.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:19.994+0000 7f279031c700 1 -- 172.21.15.7:0/947122905 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2788072fb0 con 0x7f2788100460 2023-07-20T02:45:20.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:19.994+0000 7f277f7fe700 1 -- 172.21.15.7:0/947122905 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27840039b0 con 0x7f2788100460 2023-07-20T02:45:20.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:19.994+0000 7f279031c700 1 -- 172.21.15.7:0/947122905 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2788073370 con 0x7f2788100460 2023-07-20T02:45:20.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:19.995+0000 7f277f7fe700 1 -- 172.21.15.7:0/947122905 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f2784003c70 con 0x7f2788100460 2023-07-20T02:45:20.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:19.995+0000 7f279031c700 1 -- 172.21.15.7:0/947122905 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2788061da0 con 0x7f2788100460 2023-07-20T02:45:20.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:19.996+0000 7f277f7fe700 1 --2- 172.21.15.7:0/947122905 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f27740387b0 0x7f277403ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:20.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:19.996+0000 7f277f7fe700 1 -- 172.21.15.7:0/947122905 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f278404c5e0 con 0x7f2788100460 2023-07-20T02:45:20.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:19.996+0000 7f278d8b7700 1 --2- 172.21.15.7:0/947122905 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f27740387b0 0x7f277403ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:20.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:19.997+0000 7f278d8b7700 1 --2- 172.21.15.7:0/947122905 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f27740387b0 0x7f277403ac70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2778006fd0 tx=0x7f2778006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:45:20.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:20.001+0000 7f277f7fe700 1 -- 172.21.15.7:0/947122905 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f278402a980 con 0x7f2788100460 2023-07-20T02:45:20.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:20.223+0000 7f279031c700 1 -- 172.21.15.7:0/947122905 --> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi188", "target": ["mon-mgr", ""]}) v1 -- 0x7f2788073760 con 0x7f27740387b0 2023-07-20T02:45:20.693 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:20 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:20.693 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:20 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:20.693 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:20 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:20.694 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:20 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:20.694 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:20 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi158", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:45:20.694 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:20 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:45:20.694 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:20 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:45:20.694 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:20 smithi007 ceph-mon[126353]: Updating smithi158:/etc/ceph/ceph.conf 2023-07-20T02:45:20.694 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:20 smithi007 ceph-mon[126353]: Updating smithi158:/var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T02:45:20.694 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:20 smithi007 ceph-mon[126353]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:20.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:20 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:20.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:20 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:20.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:20 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:21.693 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:21 smithi007 ceph-mon[126353]: Updating smithi158:/etc/ceph/ceph.client.admin.keyring 2023-07-20T02:45:21.693 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:21 smithi007 ceph-mon[126353]: Updating smithi158:/var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T02:45:21.694 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:21 smithi007 ceph-mon[126353]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi188", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:45:22.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:22.601+0000 7f277f7fe700 1 -- 172.21.15.7:0/947122905 <== mgr.14152 v2:172.21.15.7:6800/1354542137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f2788073760 con 0x7f27740387b0 2023-07-20T02:45:22.610 INFO:teuthology.orchestra.run.smithi007.stdout:Added host 'smithi188' with addr '172.21.15.188' 2023-07-20T02:45:22.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:22.603+0000 7f279031c700 1 -- 172.21.15.7:0/947122905 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f27740387b0 msgr2=0x7f277403ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:22.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:22.603+0000 7f279031c700 1 --2- 172.21.15.7:0/947122905 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f27740387b0 0x7f277403ac70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2778006fd0 tx=0x7f2778006e40 comp rx=0 tx=0).stop 2023-07-20T02:45:22.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:22.603+0000 7f279031c700 1 -- 172.21.15.7:0/947122905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2788100460 msgr2=0x7f2788072a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:22.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:22.603+0000 7f279031c700 1 --2- 172.21.15.7:0/947122905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2788100460 0x7f2788072a70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2784004f40 tx=0x7f2784005020 comp rx=0 tx=0).stop 2023-07-20T02:45:22.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:22.604+0000 7f279031c700 1 -- 172.21.15.7:0/947122905 shutdown_connections 2023-07-20T02:45:22.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:22.604+0000 7f279031c700 1 --2- 172.21.15.7:0/947122905 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f27740387b0 0x7f277403ac70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:22.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:22.604+0000 7f279031c700 1 --2- 172.21.15.7:0/947122905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2788100460 0x7f2788072a70 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:22.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:22.604+0000 7f279031c700 1 -- 172.21.15.7:0/947122905 >> 172.21.15.7:0/947122905 conn(0x7f27880fbd30 msgr2=0x7f2788105fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:45:22.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:22.604+0000 7f279031c700 1 -- 172.21.15.7:0/947122905 shutdown_connections 2023-07-20T02:45:22.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:22.604+0000 7f279031c700 1 -- 172.21.15.7:0/947122905 wait complete. 2023-07-20T02:45:22.635 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:22 smithi007 ceph-mon[126353]: Deploying cephadm binary to smithi188 2023-07-20T02:45:22.635 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:22 smithi007 ceph-mon[126353]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:23.127 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph orch host ls --format=json 2023-07-20T02:45:23.858 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:45:23.943 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:23 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:23.943 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:23 smithi007 ceph-mon[126353]: Added host smithi188 2023-07-20T02:45:23.943 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:23 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:45:23.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:23 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:24.693 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:24 smithi007 ceph-mon[126353]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:24.693 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:24 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:25.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.239+0000 7f83a315d700 1 -- 172.21.15.7:0/4163567345 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f839c072ba0 msgr2=0x7f839c072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:25.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.239+0000 7f83a315d700 1 --2- 172.21.15.7:0/4163567345 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f839c072ba0 0x7f839c072f80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8390009910 tx=0x7f8390009c20 comp rx=0 tx=0).stop 2023-07-20T02:45:25.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.239+0000 7f83a315d700 1 -- 172.21.15.7:0/4163567345 shutdown_connections 2023-07-20T02:45:25.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.239+0000 7f83a315d700 1 --2- 172.21.15.7:0/4163567345 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f839c072ba0 0x7f839c072f80 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:25.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.239+0000 7f83a315d700 1 -- 172.21.15.7:0/4163567345 >> 172.21.15.7:0/4163567345 conn(0x7f839c0fbd30 msgr2=0x7f839c0fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:45:25.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.239+0000 7f83a315d700 1 -- 172.21.15.7:0/4163567345 shutdown_connections 2023-07-20T02:45:25.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.239+0000 7f83a315d700 1 -- 172.21.15.7:0/4163567345 wait complete. 2023-07-20T02:45:25.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.240+0000 7f83a315d700 1 Processor -- start 2023-07-20T02:45:25.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.240+0000 7f83a315d700 1 -- start start 2023-07-20T02:45:25.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.241+0000 7f83a315d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f839c072ba0 0x7f839c19d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:25.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.241+0000 7f83a315d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839c19dc30 con 0x7f839c072ba0 2023-07-20T02:45:25.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.241+0000 7f83a0ef9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f839c072ba0 0x7f839c19d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:25.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.241+0000 7f83a0ef9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f839c072ba0 0x7f839c19d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60426/0 (socket says 172.21.15.7:60426) 2023-07-20T02:45:25.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.241+0000 7f83a0ef9700 1 -- 172.21.15.7:0/2479487822 learned_addr learned my addr 172.21.15.7:0/2479487822 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:45:25.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.241+0000 7f83a0ef9700 1 -- 172.21.15.7:0/2479487822 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83900095f0 con 0x7f839c072ba0 2023-07-20T02:45:25.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.242+0000 7f83a0ef9700 1 --2- 172.21.15.7:0/2479487822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f839c072ba0 0x7f839c19d6f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8390006010 tx=0x7f8390004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:45:25.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.242+0000 7f8399ffb700 1 -- 172.21.15.7:0/2479487822 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f839001d070 con 0x7f839c072ba0 2023-07-20T02:45:25.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.242+0000 7f8399ffb700 1 -- 172.21.15.7:0/2479487822 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f83900044a0 con 0x7f839c072ba0 2023-07-20T02:45:25.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.242+0000 7f83a315d700 1 -- 172.21.15.7:0/2479487822 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f839c19ddd0 con 0x7f839c072ba0 2023-07-20T02:45:25.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.242+0000 7f8399ffb700 1 -- 172.21.15.7:0/2479487822 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8390003a70 con 0x7f839c072ba0 2023-07-20T02:45:25.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.242+0000 7f83a315d700 1 -- 172.21.15.7:0/2479487822 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f839c19e1f0 con 0x7f839c072ba0 2023-07-20T02:45:25.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.243+0000 7f8399ffb700 1 -- 172.21.15.7:0/2479487822 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f8390003cb0 con 0x7f839c072ba0 2023-07-20T02:45:25.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.243+0000 7f83a315d700 1 -- 172.21.15.7:0/2479487822 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f839c061da0 con 0x7f839c072ba0 2023-07-20T02:45:25.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.244+0000 7f8399ffb700 1 --2- 172.21.15.7:0/2479487822 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f838c0387b0 0x7f838c03ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:25.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.244+0000 7f8399ffb700 1 -- 172.21.15.7:0/2479487822 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f839004c690 con 0x7f839c072ba0 2023-07-20T02:45:25.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.244+0000 7f839bfff700 1 --2- 172.21.15.7:0/2479487822 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f838c0387b0 0x7f838c03ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:25.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.244+0000 7f839bfff700 1 --2- 172.21.15.7:0/2479487822 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f838c0387b0 0x7f838c03ac70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8388006fd0 tx=0x7f8388006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:45:25.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.249+0000 7f8399ffb700 1 -- 172.21.15.7:0/2479487822 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8390029540 con 0x7f839c072ba0 2023-07-20T02:45:25.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.460+0000 7f83a315d700 1 -- 172.21.15.7:0/2479487822 --> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f839c1097c0 con 0x7f838c0387b0 2023-07-20T02:45:25.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.461+0000 7f8399ffb700 1 -- 172.21.15.7:0/2479487822 <== mgr.14152 v2:172.21.15.7:6800/1354542137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+239 (secure 0 0 0) 0x7f839c1097c0 con 0x7f838c0387b0 2023-07-20T02:45:25.469 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:45:25.470 INFO:teuthology.orchestra.run.smithi007.stdout:[{"addr": "172.21.15.7", "hostname": "smithi007", "labels": [], "status": ""}, {"addr": "172.21.15.158", "hostname": "smithi158", "labels": [], "status": ""}, {"addr": "172.21.15.188", "hostname": "smithi188", "labels": [], "status": ""}] 2023-07-20T02:45:25.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.463+0000 7f83a315d700 1 -- 172.21.15.7:0/2479487822 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f838c0387b0 msgr2=0x7f838c03ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:25.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.463+0000 7f83a315d700 1 --2- 172.21.15.7:0/2479487822 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f838c0387b0 0x7f838c03ac70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8388006fd0 tx=0x7f8388006e40 comp rx=0 tx=0).stop 2023-07-20T02:45:25.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.463+0000 7f83a315d700 1 -- 172.21.15.7:0/2479487822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f839c072ba0 msgr2=0x7f839c19d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:25.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.463+0000 7f83a315d700 1 --2- 172.21.15.7:0/2479487822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f839c072ba0 0x7f839c19d6f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8390006010 tx=0x7f8390004ff0 comp rx=0 tx=0).stop 2023-07-20T02:45:25.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.463+0000 7f83a315d700 1 -- 172.21.15.7:0/2479487822 shutdown_connections 2023-07-20T02:45:25.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.463+0000 7f83a315d700 1 --2- 172.21.15.7:0/2479487822 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f838c0387b0 0x7f838c03ac70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:25.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.463+0000 7f83a315d700 1 --2- 172.21.15.7:0/2479487822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f839c072ba0 0x7f839c19d6f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:25.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.463+0000 7f83a315d700 1 -- 172.21.15.7:0/2479487822 >> 172.21.15.7:0/2479487822 conn(0x7f839c0fbd30 msgr2=0x7f839c106d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:45:25.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.463+0000 7f83a315d700 1 -- 172.21.15.7:0/2479487822 shutdown_connections 2023-07-20T02:45:25.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:25.463+0000 7f83a315d700 1 -- 172.21.15.7:0/2479487822 wait complete. 2023-07-20T02:45:25.938 INFO:tasks.cephadm:Setting crush tunables to default 2023-07-20T02:45:25.939 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph osd crush tunables default 2023-07-20T02:45:26.193 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:25 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:26.193 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:25 smithi007 ceph-mon[126353]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-20T02:45:26.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:25 smithi007 ceph-mon[126353]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:26.666 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:45:28.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.159+0000 7faa25de7700 1 -- 172.21.15.7:0/671023209 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa20100460 msgr2=0x7faa20100840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:28.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.159+0000 7faa25de7700 1 --2- 172.21.15.7:0/671023209 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa20100460 0x7faa20100840 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7faa10009910 tx=0x7faa10009c20 comp rx=0 tx=0).stop 2023-07-20T02:45:28.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.159+0000 7faa25de7700 1 -- 172.21.15.7:0/671023209 shutdown_connections 2023-07-20T02:45:28.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.159+0000 7faa25de7700 1 --2- 172.21.15.7:0/671023209 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa20100460 0x7faa20100840 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:28.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.159+0000 7faa25de7700 1 -- 172.21.15.7:0/671023209 >> 172.21.15.7:0/671023209 conn(0x7faa200fbd30 msgr2=0x7faa200fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:45:28.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.159+0000 7faa25de7700 1 -- 172.21.15.7:0/671023209 shutdown_connections 2023-07-20T02:45:28.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.159+0000 7faa25de7700 1 -- 172.21.15.7:0/671023209 wait complete. 2023-07-20T02:45:28.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.161+0000 7faa25de7700 1 Processor -- start 2023-07-20T02:45:28.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.161+0000 7faa25de7700 1 -- start start 2023-07-20T02:45:28.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.162+0000 7faa25de7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa20100460 0x7faa20072a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:28.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.162+0000 7faa25de7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa20074950 con 0x7faa20100460 2023-07-20T02:45:28.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.162+0000 7faa1f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa20100460 0x7faa20072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:28.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.162+0000 7faa1f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa20100460 0x7faa20072a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60454/0 (socket says 172.21.15.7:60454) 2023-07-20T02:45:28.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.162+0000 7faa1f7fe700 1 -- 172.21.15.7:0/2606875359 learned_addr learned my addr 172.21.15.7:0/2606875359 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:45:28.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.162+0000 7faa1f7fe700 1 -- 172.21.15.7:0/2606875359 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa100095f0 con 0x7faa20100460 2023-07-20T02:45:28.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.163+0000 7faa1f7fe700 1 --2- 172.21.15.7:0/2606875359 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa20100460 0x7faa20072a70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7faa10006010 tx=0x7faa10004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:45:28.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.163+0000 7faa1cff9700 1 -- 172.21.15.7:0/2606875359 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faa1001d070 con 0x7faa20100460 2023-07-20T02:45:28.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.163+0000 7faa1cff9700 1 -- 172.21.15.7:0/2606875359 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faa1000bc10 con 0x7faa20100460 2023-07-20T02:45:28.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.163+0000 7faa1cff9700 1 -- 172.21.15.7:0/2606875359 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faa10003910 con 0x7faa20100460 2023-07-20T02:45:28.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.163+0000 7faa25de7700 1 -- 172.21.15.7:0/2606875359 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa20072fb0 con 0x7faa20100460 2023-07-20T02:45:28.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.163+0000 7faa25de7700 1 -- 172.21.15.7:0/2606875359 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa20073370 con 0x7faa20100460 2023-07-20T02:45:28.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.164+0000 7faa1cff9700 1 -- 172.21.15.7:0/2606875359 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7faa1000bd80 con 0x7faa20100460 2023-07-20T02:45:28.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.164+0000 7faa25de7700 1 -- 172.21.15.7:0/2606875359 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa20061da0 con 0x7faa20100460 2023-07-20T02:45:28.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.165+0000 7faa1cff9700 1 --2- 172.21.15.7:0/2606875359 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7faa08038800 0x7faa0803acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:28.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.165+0000 7faa1cff9700 1 -- 172.21.15.7:0/2606875359 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7faa1004c5a0 con 0x7faa20100460 2023-07-20T02:45:28.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.165+0000 7faa1effd700 1 --2- 172.21.15.7:0/2606875359 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7faa08038800 0x7faa0803acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:28.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.166+0000 7faa1effd700 1 --2- 172.21.15.7:0/2606875359 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7faa08038800 0x7faa0803acc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7faa14006fd0 tx=0x7faa14006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:45:28.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.170+0000 7faa1cff9700 1 -- 172.21.15.7:0/2606875359 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7faa10029360 con 0x7faa20100460 2023-07-20T02:45:28.193 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:27 smithi007 ceph-mon[126353]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:28.193 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:27 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:28.193 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:27 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:28.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:27 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:28.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:27 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:28.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:27 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi188", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:45:28.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:27 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:45:28.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:27 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:45:28.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.383+0000 7faa25de7700 1 -- 172.21.15.7:0/2606875359 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7faa20073710 con 0x7faa20100460 2023-07-20T02:45:28.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.799+0000 7faa1cff9700 1 -- 172.21.15.7:0/2606875359 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7faa10026020 con 0x7faa20100460 2023-07-20T02:45:28.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.802+0000 7faa25de7700 1 -- 172.21.15.7:0/2606875359 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7faa08038800 msgr2=0x7faa0803acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:28.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.802+0000 7faa25de7700 1 --2- 172.21.15.7:0/2606875359 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7faa08038800 0x7faa0803acc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7faa14006fd0 tx=0x7faa14006e40 comp rx=0 tx=0).stop 2023-07-20T02:45:28.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.802+0000 7faa25de7700 1 -- 172.21.15.7:0/2606875359 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa20100460 msgr2=0x7faa20072a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:28.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.802+0000 7faa25de7700 1 --2- 172.21.15.7:0/2606875359 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa20100460 0x7faa20072a70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7faa10006010 tx=0x7faa10004ff0 comp rx=0 tx=0).stop 2023-07-20T02:45:28.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.802+0000 7faa25de7700 1 -- 172.21.15.7:0/2606875359 shutdown_connections 2023-07-20T02:45:28.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.802+0000 7faa25de7700 1 --2- 172.21.15.7:0/2606875359 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7faa08038800 0x7faa0803acc0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:28.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.802+0000 7faa25de7700 1 --2- 172.21.15.7:0/2606875359 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa20100460 0x7faa20072a70 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:28.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.802+0000 7faa25de7700 1 -- 172.21.15.7:0/2606875359 >> 172.21.15.7:0/2606875359 conn(0x7faa200fbd30 msgr2=0x7faa20105fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:45:28.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.802+0000 7faa25de7700 1 -- 172.21.15.7:0/2606875359 shutdown_connections 2023-07-20T02:45:28.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:28.802+0000 7faa25de7700 1 -- 172.21.15.7:0/2606875359 wait complete. 2023-07-20T02:45:28.813 INFO:teuthology.orchestra.run.smithi007.stderr:adjusted tunables profile to default 2023-07-20T02:45:29.193 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:28 smithi007 ceph-mon[126353]: Updating smithi188:/etc/ceph/ceph.conf 2023-07-20T02:45:29.193 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:28 smithi007 ceph-mon[126353]: Updating smithi188:/var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T02:45:29.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:28 smithi007 ceph-mon[126353]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2023-07-20T02:45:29.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:28 smithi007 ceph-mon[126353]: Updating smithi188:/var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T02:45:29.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:28 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/2606875359' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-20T02:45:29.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:28 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:29.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:28 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:29.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:28 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:29.250 INFO:tasks.cephadm:Adding mon.a on smithi007 2023-07-20T02:45:29.251 INFO:tasks.cephadm:Adding mon.b on smithi158 2023-07-20T02:45:29.251 INFO:tasks.cephadm:Adding mon.c on smithi188 2023-07-20T02:45:29.251 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph orch apply mon '3;smithi007:172.21.15.7=a;smithi158:172.21.15.158=b;smithi188:172.21.15.188=c' 2023-07-20T02:45:29.833 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T02:45:30.193 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:29 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/2606875359' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-20T02:45:30.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:29 smithi007 ceph-mon[126353]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T02:45:30.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:29 smithi007 ceph-mon[126353]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:31.142 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.132+0000 7f8a32ca0700 1 -- 172.21.15.188:0/1512919894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a2c0ff340 msgr2=0x7f8a2c0ff720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:31.142 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.132+0000 7f8a32ca0700 1 --2- 172.21.15.188:0/1512919894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a2c0ff340 0x7f8a2c0ff720 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8a20009910 tx=0x7f8a20009c20 comp rx=0 tx=0).stop 2023-07-20T02:45:31.142 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.132+0000 7f8a32ca0700 1 -- 172.21.15.188:0/1512919894 shutdown_connections 2023-07-20T02:45:31.143 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.132+0000 7f8a32ca0700 1 --2- 172.21.15.188:0/1512919894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a2c0ff340 0x7f8a2c0ff720 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:31.143 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.132+0000 7f8a32ca0700 1 -- 172.21.15.188:0/1512919894 >> 172.21.15.188:0/1512919894 conn(0x7f8a2c0fabf0 msgr2=0x7f8a2c0fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:45:31.143 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.132+0000 7f8a32ca0700 1 -- 172.21.15.188:0/1512919894 shutdown_connections 2023-07-20T02:45:31.143 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.132+0000 7f8a32ca0700 1 -- 172.21.15.188:0/1512919894 wait complete. 2023-07-20T02:45:31.143 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.133+0000 7f8a32ca0700 1 Processor -- start 2023-07-20T02:45:31.143 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.133+0000 7f8a32ca0700 1 -- start start 2023-07-20T02:45:31.143 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.133+0000 7f8a32ca0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a2c0ff340 0x7f8a2c196550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:31.144 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.133+0000 7f8a32ca0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2c196a90 con 0x7f8a2c0ff340 2023-07-20T02:45:31.145 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.134+0000 7f8a30a3c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a2c0ff340 0x7f8a2c196550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:31.145 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.134+0000 7f8a30a3c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a2c0ff340 0x7f8a2c196550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.188:56532/0 (socket says 172.21.15.188:56532) 2023-07-20T02:45:31.145 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.134+0000 7f8a30a3c700 1 -- 172.21.15.188:0/3485196803 learned_addr learned my addr 172.21.15.188:0/3485196803 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-07-20T02:45:31.145 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.134+0000 7f8a30a3c700 1 -- 172.21.15.188:0/3485196803 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a200095f0 con 0x7f8a2c0ff340 2023-07-20T02:45:31.145 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.134+0000 7f8a30a3c700 1 --2- 172.21.15.188:0/3485196803 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a2c0ff340 0x7f8a2c196550 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8a20004f40 tx=0x7f8a20005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:45:31.146 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.135+0000 7f8a29ffb700 1 -- 172.21.15.188:0/3485196803 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a2001d070 con 0x7f8a2c0ff340 2023-07-20T02:45:31.146 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.135+0000 7f8a32ca0700 1 -- 172.21.15.188:0/3485196803 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a2c196c90 con 0x7f8a2c0ff340 2023-07-20T02:45:31.146 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.135+0000 7f8a29ffb700 1 -- 172.21.15.188:0/3485196803 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a20004370 con 0x7f8a2c0ff340 2023-07-20T02:45:31.146 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.135+0000 7f8a29ffb700 1 -- 172.21.15.188:0/3485196803 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a200039b0 con 0x7f8a2c0ff340 2023-07-20T02:45:31.146 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.135+0000 7f8a32ca0700 1 -- 172.21.15.188:0/3485196803 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a2c1970b0 con 0x7f8a2c0ff340 2023-07-20T02:45:31.146 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.136+0000 7f8a32ca0700 1 -- 172.21.15.188:0/3485196803 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a2c107650 con 0x7f8a2c0ff340 2023-07-20T02:45:31.146 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.136+0000 7f8a29ffb700 1 -- 172.21.15.188:0/3485196803 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f8a20003c70 con 0x7f8a2c0ff340 2023-07-20T02:45:31.147 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.136+0000 7f8a29ffb700 1 --2- 172.21.15.188:0/3485196803 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f8a1c0387b0 0x7f8a1c03ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:31.147 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.136+0000 7f8a29ffb700 1 -- 172.21.15.188:0/3485196803 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8a2004c520 con 0x7f8a2c0ff340 2023-07-20T02:45:31.147 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.136+0000 7f8a2bfff700 1 --2- 172.21.15.188:0/3485196803 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f8a1c0387b0 0x7f8a1c03ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:31.147 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.137+0000 7f8a2bfff700 1 --2- 172.21.15.188:0/3485196803 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f8a1c0387b0 0x7f8a1c03ac70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8a18006fd0 tx=0x7f8a18006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:45:31.150 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.141+0000 7f8a29ffb700 1 -- 172.21.15.188:0/3485196803 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8a2002a980 con 0x7f8a2c0ff340 2023-07-20T02:45:31.343 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.334+0000 7f8a32ca0700 1 -- 172.21.15.188:0/3485196803 --> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi007:172.21.15.7=a;smithi158:172.21.15.158=b;smithi188:172.21.15.188=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f8a2c0688b0 con 0x7f8a1c0387b0 2023-07-20T02:45:31.350 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.342+0000 7f8a29ffb700 1 -- 172.21.15.188:0/3485196803 <== mgr.14152 v2:172.21.15.7:6800/1354542137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f8a2c0688b0 con 0x7f8a1c0387b0 2023-07-20T02:45:31.351 INFO:teuthology.orchestra.run.smithi188.stdout:Scheduled mon update... 2023-07-20T02:45:31.353 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.344+0000 7f8a32ca0700 1 -- 172.21.15.188:0/3485196803 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f8a1c0387b0 msgr2=0x7f8a1c03ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:31.353 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.344+0000 7f8a32ca0700 1 --2- 172.21.15.188:0/3485196803 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f8a1c0387b0 0x7f8a1c03ac70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8a18006fd0 tx=0x7f8a18006e40 comp rx=0 tx=0).stop 2023-07-20T02:45:31.353 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.344+0000 7f8a32ca0700 1 -- 172.21.15.188:0/3485196803 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a2c0ff340 msgr2=0x7f8a2c196550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:31.353 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.344+0000 7f8a32ca0700 1 --2- 172.21.15.188:0/3485196803 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a2c0ff340 0x7f8a2c196550 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8a20004f40 tx=0x7f8a20005020 comp rx=0 tx=0).stop 2023-07-20T02:45:31.353 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.344+0000 7f8a32ca0700 1 -- 172.21.15.188:0/3485196803 shutdown_connections 2023-07-20T02:45:31.354 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.344+0000 7f8a32ca0700 1 --2- 172.21.15.188:0/3485196803 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f8a1c0387b0 0x7f8a1c03ac70 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:31.354 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.344+0000 7f8a32ca0700 1 --2- 172.21.15.188:0/3485196803 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a2c0ff340 0x7f8a2c196550 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:31.354 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.344+0000 7f8a32ca0700 1 -- 172.21.15.188:0/3485196803 >> 172.21.15.188:0/3485196803 conn(0x7f8a2c0fabf0 msgr2=0x7f8a2c103610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:45:31.354 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.344+0000 7f8a32ca0700 1 -- 172.21.15.188:0/3485196803 shutdown_connections 2023-07-20T02:45:31.354 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:31.344+0000 7f8a32ca0700 1 -- 172.21.15.188:0/3485196803 wait complete. 2023-07-20T02:45:31.796 DEBUG:teuthology.orchestra.run.smithi158:mon.b> sudo journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mon.b.service 2023-07-20T02:45:31.799 DEBUG:teuthology.orchestra.run.smithi188:mon.c> sudo journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mon.c.service 2023-07-20T02:45:31.801 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-20T02:45:31.801 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph mon dump -f json 2023-07-20T02:45:31.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:-- Logs begin at Thu 2023-07-20 02:09:43 UTC. -- 2023-07-20T02:45:31.834 INFO:journalctl@ceph.mon.b.smithi158.stdout:-- Logs begin at Thu 2023-07-20 02:10:12 UTC. -- 2023-07-20T02:45:32.694 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:32 smithi007 ceph-mon[126353]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi007:172.21.15.7=a;smithi158:172.21.15.158=b;smithi188:172.21.15.188=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:45:32.694 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:32 smithi007 ceph-mon[126353]: Saving service mon spec with placement smithi007:172.21.15.7=a;smithi158:172.21.15.158=b;smithi188:172.21.15.188=c;count:3 2023-07-20T02:45:32.694 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:32 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:32.694 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:32 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:45:32.694 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:32 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:45:32.694 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:32 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:45:32.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:32 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:32.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:32 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T02:45:32.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:32 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:45:32.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:32 smithi007 ceph-mon[126353]: Deploying daemon mon.c on smithi188 2023-07-20T02:45:32.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:32 smithi007 ceph-mon[126353]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:32.809 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T02:45:34.675 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.666+0000 7f9593f7c700 1 -- 172.21.15.188:0/2593389506 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f958c0679a0 msgr2=0x7f958c067d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:34.675 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.666+0000 7f9593f7c700 1 --2- 172.21.15.188:0/2593389506 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f958c0679a0 0x7f958c067d80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9588009910 tx=0x7f9588009c20 comp rx=0 tx=0).stop 2023-07-20T02:45:34.676 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.666+0000 7f9593f7c700 1 -- 172.21.15.188:0/2593389506 shutdown_connections 2023-07-20T02:45:34.676 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.666+0000 7f9593f7c700 1 --2- 172.21.15.188:0/2593389506 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f958c0679a0 0x7f958c067d80 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:34.676 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.666+0000 7f9593f7c700 1 -- 172.21.15.188:0/2593389506 >> 172.21.15.188:0/2593389506 conn(0x7f958c074c70 msgr2=0x7f958c075080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:45:34.676 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.666+0000 7f9593f7c700 1 -- 172.21.15.188:0/2593389506 shutdown_connections 2023-07-20T02:45:34.676 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.666+0000 7f9593f7c700 1 -- 172.21.15.188:0/2593389506 wait complete. 2023-07-20T02:45:34.677 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.667+0000 7f9593f7c700 1 Processor -- start 2023-07-20T02:45:34.678 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.667+0000 7f9593f7c700 1 -- start start 2023-07-20T02:45:34.678 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.667+0000 7f9593f7c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f958c0679a0 0x7f958c19d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:34.679 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.667+0000 7f9593f7c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958c19dbf0 con 0x7f958c0679a0 2023-07-20T02:45:34.679 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.667+0000 7f9591d18700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f958c0679a0 0x7f958c19d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:34.679 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.667+0000 7f9591d18700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f958c0679a0 0x7f958c19d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.188:56538/0 (socket says 172.21.15.188:56538) 2023-07-20T02:45:34.679 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.667+0000 7f9591d18700 1 -- 172.21.15.188:0/963633529 learned_addr learned my addr 172.21.15.188:0/963633529 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-07-20T02:45:34.679 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.668+0000 7f9591d18700 1 -- 172.21.15.188:0/963633529 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95880095f0 con 0x7f958c0679a0 2023-07-20T02:45:34.679 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.668+0000 7f9591d18700 1 --2- 172.21.15.188:0/963633529 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f958c0679a0 0x7f958c19d6b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9588004f40 tx=0x7f9588005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:45:34.679 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.668+0000 7f9582ffd700 1 -- 172.21.15.188:0/963633529 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f958801d070 con 0x7f958c0679a0 2023-07-20T02:45:34.679 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.668+0000 7f9593f7c700 1 -- 172.21.15.188:0/963633529 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f958c19dd90 con 0x7f958c0679a0 2023-07-20T02:45:34.680 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.668+0000 7f9582ffd700 1 -- 172.21.15.188:0/963633529 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9588004370 con 0x7f958c0679a0 2023-07-20T02:45:34.680 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.668+0000 7f9582ffd700 1 -- 172.21.15.188:0/963633529 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f95880039b0 con 0x7f958c0679a0 2023-07-20T02:45:34.680 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.668+0000 7f9593f7c700 1 -- 172.21.15.188:0/963633529 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f958c19e1b0 con 0x7f958c0679a0 2023-07-20T02:45:34.681 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.669+0000 7f9593f7c700 1 -- 172.21.15.188:0/963633529 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f958c061da0 con 0x7f958c0679a0 2023-07-20T02:45:34.681 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.669+0000 7f9582ffd700 1 -- 172.21.15.188:0/963633529 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f9588003c70 con 0x7f958c0679a0 2023-07-20T02:45:34.681 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.669+0000 7f9582ffd700 1 --2- 172.21.15.188:0/963633529 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f95780387b0 0x7f957803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:34.681 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.669+0000 7f9582ffd700 1 -- 172.21.15.188:0/963633529 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f958804c760 con 0x7f958c0679a0 2023-07-20T02:45:34.682 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.670+0000 7f9591517700 1 --2- 172.21.15.188:0/963633529 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f95780387b0 0x7f957803ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:34.682 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.670+0000 7f9591517700 1 --2- 172.21.15.188:0/963633529 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f95780387b0 0x7f957803ac70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f957c006fd0 tx=0x7f957c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:45:34.684 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.675+0000 7f9582ffd700 1 -- 172.21.15.188:0/963633529 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9588029770 con 0x7f958c0679a0 2023-07-20T02:45:34.943 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:34 smithi007 ceph-mon[126353]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:34.959 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:34 smithi188 systemd[1]: Starting Ceph mon.c for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:45:34.970 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.961+0000 7f9593f7c700 1 -- 172.21.15.188:0/963633529 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f958c198c40 con 0x7f958c0679a0 2023-07-20T02:45:34.971 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.961+0000 7f9582ffd700 1 -- 172.21.15.188:0/963633529 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+732 (secure 0 0 0) 0x7f9588026030 con 0x7f958c0679a0 2023-07-20T02:45:34.972 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-07-20T02:45:34.972 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"1b04643c-26a7-11ee-9b34-001a4aab830c","modified":"2023-07-20T02:43:46.393290Z","created":"2023-07-20T02:43:46.393290Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3300","nonce":0},{"type":"v1","addr":"172.21.15.7:6789","nonce":0}]},"addr":"172.21.15.7:6789/0","public_addr":"172.21.15.7:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-07-20T02:45:34.972 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.964+0000 7f9593f7c700 1 -- 172.21.15.188:0/963633529 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f95780387b0 msgr2=0x7f957803ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:34.973 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.964+0000 7f9593f7c700 1 --2- 172.21.15.188:0/963633529 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f95780387b0 0x7f957803ac70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f957c006fd0 tx=0x7f957c006e40 comp rx=0 tx=0).stop 2023-07-20T02:45:34.973 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.964+0000 7f9593f7c700 1 -- 172.21.15.188:0/963633529 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f958c0679a0 msgr2=0x7f958c19d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:34.973 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.964+0000 7f9593f7c700 1 --2- 172.21.15.188:0/963633529 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f958c0679a0 0x7f958c19d6b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9588004f40 tx=0x7f9588005020 comp rx=0 tx=0).stop 2023-07-20T02:45:34.973 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.964+0000 7f9593f7c700 1 -- 172.21.15.188:0/963633529 shutdown_connections 2023-07-20T02:45:34.973 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.964+0000 7f9593f7c700 1 --2- 172.21.15.188:0/963633529 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f95780387b0 0x7f957803ac70 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:34.973 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.964+0000 7f9593f7c700 1 --2- 172.21.15.188:0/963633529 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f958c0679a0 0x7f958c19d6b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:34.974 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.964+0000 7f9593f7c700 1 -- 172.21.15.188:0/963633529 >> 172.21.15.188:0/963633529 conn(0x7f958c074c70 msgr2=0x7f958c0fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:45:34.974 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.964+0000 7f9593f7c700 1 -- 172.21.15.188:0/963633529 shutdown_connections 2023-07-20T02:45:34.974 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:34.964+0000 7f9593f7c700 1 -- 172.21.15.188:0/963633529 wait complete. 2023-07-20T02:45:34.974 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2023-07-20T02:45:35.459 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 podman[129487]: 2023-07-20T02:45:35.943 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:35 smithi007 ceph-mon[126353]: from='client.? 172.21.15.188:0/963633529' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T02:45:35.959 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: set uid:gid to 167:167 (ceph:ceph) 2023-07-20T02:45:35.959 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev), process ceph-mon, pid 7 2023-07-20T02:45:35.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: pidfile_write: ignore empty --pid-file 2023-07-20T02:45:35.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: load: jerasure load: lrc 2023-07-20T02:45:35.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: RocksDB version: 7.9.2 2023-07-20T02:45:35.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Git sha 0 2023-07-20T02:45:35.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Compile date 2023-07-19 21:36:35 2023-07-20T02:45:35.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: DB SUMMARY 2023-07-20T02:45:35.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: DB Session ID: WL1W0VLCF37XUW4XWB70 2023-07-20T02:45:35.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: CURRENT file: CURRENT 2023-07-20T02:45:35.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: IDENTITY file: IDENTITY 2023-07-20T02:45:35.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-07-20T02:45:35.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2023-07-20T02:45:35.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2023-07-20T02:45:35.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.error_if_exists: 0 2023-07-20T02:45:35.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.create_if_missing: 0 2023-07-20T02:45:35.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.paranoid_checks: 1 2023-07-20T02:45:35.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-20T02:45:35.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-20T02:45:35.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-20T02:45:35.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.env: 0x55848df51c60 2023-07-20T02:45:35.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.fs: PosixFileSystem 2023-07-20T02:45:35.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.info_log: 0x55848fcf32c0 2023-07-20T02:45:35.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.max_file_opening_threads: 16 2023-07-20T02:45:35.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.statistics: (nil) 2023-07-20T02:45:35.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.use_fsync: 0 2023-07-20T02:45:35.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.max_log_file_size: 0 2023-07-20T02:45:35.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-20T02:45:35.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-20T02:45:35.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.keep_log_file_num: 1000 2023-07-20T02:45:35.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.recycle_log_file_num: 0 2023-07-20T02:45:35.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.allow_fallocate: 1 2023-07-20T02:45:35.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.allow_mmap_reads: 0 2023-07-20T02:45:35.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.allow_mmap_writes: 0 2023-07-20T02:45:35.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.use_direct_reads: 0 2023-07-20T02:45:35.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-20T02:45:35.965 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.create_missing_column_families: 0 2023-07-20T02:45:35.965 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.db_log_dir: 2023-07-20T02:45:35.965 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.wal_dir: 2023-07-20T02:45:35.965 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-20T02:45:35.965 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-20T02:45:35.965 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-20T02:45:35.965 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-20T02:45:35.966 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-20T02:45:35.966 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-20T02:45:35.966 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.advise_random_on_open: 1 2023-07-20T02:45:35.966 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.db_write_buffer_size: 0 2023-07-20T02:45:35.966 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.write_buffer_manager: 0x55848fc143c0 2023-07-20T02:45:35.966 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-20T02:45:35.966 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-20T02:45:35.967 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-20T02:45:35.967 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.rate_limiter: (nil) 2023-07-20T02:45:35.967 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-20T02:45:35.967 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.wal_recovery_mode: 2 2023-07-20T02:45:35.967 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.enable_thread_tracking: 0 2023-07-20T02:45:35.967 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.enable_pipelined_write: 0 2023-07-20T02:45:35.967 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.unordered_write: 0 2023-07-20T02:45:35.967 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-20T02:45:35.968 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-20T02:45:35.968 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-20T02:45:35.968 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-20T02:45:35.968 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.row_cache: None 2023-07-20T02:45:35.968 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.wal_filter: None 2023-07-20T02:45:35.968 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-20T02:45:35.968 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.allow_ingest_behind: 0 2023-07-20T02:45:35.969 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.two_write_queues: 0 2023-07-20T02:45:35.970 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.manual_wal_flush: 0 2023-07-20T02:45:35.970 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.wal_compression: 0 2023-07-20T02:45:35.970 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.atomic_flush: 0 2023-07-20T02:45:35.970 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-20T02:45:35.970 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-20T02:45:35.970 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-20T02:45:35.970 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.log_readahead_size: 0 2023-07-20T02:45:35.971 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-20T02:45:35.971 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.best_efforts_recovery: 0 2023-07-20T02:45:35.971 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-20T02:45:35.971 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-20T02:45:35.971 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.allow_data_in_errors: 0 2023-07-20T02:45:35.971 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.db_host_id: __hostname__ 2023-07-20T02:45:35.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-20T02:45:35.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.max_background_jobs: 2 2023-07-20T02:45:35.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.max_background_compactions: -1 2023-07-20T02:45:35.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.max_subcompactions: 1 2023-07-20T02:45:35.973 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-20T02:45:35.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-20T02:45:35.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-20T02:45:35.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.max_total_wal_size: 0 2023-07-20T02:45:35.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-20T02:45:35.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-20T02:45:35.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-20T02:45:35.974 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-20T02:45:35.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.max_open_files: -1 2023-07-20T02:45:35.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.bytes_per_sync: 0 2023-07-20T02:45:35.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-20T02:45:35.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-20T02:45:35.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.compaction_readahead_size: 0 2023-07-20T02:45:35.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.max_background_flushes: -1 2023-07-20T02:45:35.975 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Compression algorithms supported: 2023-07-20T02:45:35.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-20T02:45:35.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: kZSTD supported: 0 2023-07-20T02:45:35.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: kXpressCompression supported: 0 2023-07-20T02:45:35.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: kLZ4HCCompression supported: 1 2023-07-20T02:45:35.976 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: kZlibCompression supported: 1 2023-07-20T02:45:35.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: kSnappyCompression supported: 1 2023-07-20T02:45:35.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: kLZ4Compression supported: 1 2023-07-20T02:45:35.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: kBZip2Compression supported: 0 2023-07-20T02:45:35.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-20T02:45:35.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-20T02:45:35.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2023-07-20T02:45:35.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-20T02:45:35.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-20T02:45:35.978 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.merge_operator: 2023-07-20T02:45:35.978 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.compaction_filter: None 2023-07-20T02:45:35.978 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.compaction_filter_factory: None 2023-07-20T02:45:35.978 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.sst_partitioner_factory: None 2023-07-20T02:45:35.978 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-20T02:45:35.978 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-20T02:45:35.978 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55848fcf33c0) 2023-07-20T02:45:35.979 INFO:journalctl@ceph.mon.c.smithi188.stdout: cache_index_and_filter_blocks: 1 2023-07-20T02:45:35.979 INFO:journalctl@ceph.mon.c.smithi188.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-20T02:45:35.979 INFO:journalctl@ceph.mon.c.smithi188.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-20T02:45:35.979 INFO:journalctl@ceph.mon.c.smithi188.stdout: pin_top_level_index_and_filter: 1 2023-07-20T02:45:35.979 INFO:journalctl@ceph.mon.c.smithi188.stdout: index_type: 0 2023-07-20T02:45:35.979 INFO:journalctl@ceph.mon.c.smithi188.stdout: data_block_index_type: 0 2023-07-20T02:45:35.979 INFO:journalctl@ceph.mon.c.smithi188.stdout: index_shortening: 1 2023-07-20T02:45:35.980 INFO:journalctl@ceph.mon.c.smithi188.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-20T02:45:35.980 INFO:journalctl@ceph.mon.c.smithi188.stdout: checksum: 4 2023-07-20T02:45:35.980 INFO:journalctl@ceph.mon.c.smithi188.stdout: no_block_cache: 0 2023-07-20T02:45:35.980 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_cache: 0x55848fcb3350 2023-07-20T02:45:35.980 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_cache_name: BinnedLRUCache 2023-07-20T02:45:35.980 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_cache_options: 2023-07-20T02:45:35.980 INFO:journalctl@ceph.mon.c.smithi188.stdout: capacity : 536870912 2023-07-20T02:45:35.981 INFO:journalctl@ceph.mon.c.smithi188.stdout: num_shard_bits : 4 2023-07-20T02:45:35.981 INFO:journalctl@ceph.mon.c.smithi188.stdout: strict_capacity_limit : 0 2023-07-20T02:45:35.981 INFO:journalctl@ceph.mon.c.smithi188.stdout: high_pri_pool_ratio: 0.000 2023-07-20T02:45:35.981 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_cache_compressed: (nil) 2023-07-20T02:45:35.981 INFO:journalctl@ceph.mon.c.smithi188.stdout: persistent_cache: (nil) 2023-07-20T02:45:35.981 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_size: 4096 2023-07-20T02:45:35.981 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_size_deviation: 10 2023-07-20T02:45:35.982 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_restart_interval: 16 2023-07-20T02:45:35.982 INFO:journalctl@ceph.mon.c.smithi188.stdout: index_block_restart_interval: 1 2023-07-20T02:45:35.982 INFO:journalctl@ceph.mon.c.smithi188.stdout: metadata_block_size: 4096 2023-07-20T02:45:35.982 INFO:journalctl@ceph.mon.c.smithi188.stdout: partition_filters: 0 2023-07-20T02:45:35.982 INFO:journalctl@ceph.mon.c.smithi188.stdout: use_delta_encoding: 1 2023-07-20T02:45:35.982 INFO:journalctl@ceph.mon.c.smithi188.stdout: filter_policy: bloomfilter 2023-07-20T02:45:35.982 INFO:journalctl@ceph.mon.c.smithi188.stdout: whole_key_filtering: 1 2023-07-20T02:45:35.982 INFO:journalctl@ceph.mon.c.smithi188.stdout: verify_compression: 0 2023-07-20T02:45:35.983 INFO:journalctl@ceph.mon.c.smithi188.stdout: read_amp_bytes_per_bit: 0 2023-07-20T02:45:35.983 INFO:journalctl@ceph.mon.c.smithi188.stdout: format_version: 5 2023-07-20T02:45:35.983 INFO:journalctl@ceph.mon.c.smithi188.stdout: enable_index_compression: 1 2023-07-20T02:45:35.983 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_align: 0 2023-07-20T02:45:35.983 INFO:journalctl@ceph.mon.c.smithi188.stdout: max_auto_readahead_size: 262144 2023-07-20T02:45:35.983 INFO:journalctl@ceph.mon.c.smithi188.stdout: prepopulate_block_cache: 0 2023-07-20T02:45:35.983 INFO:journalctl@ceph.mon.c.smithi188.stdout: initial_auto_readahead_size: 8192 2023-07-20T02:45:35.984 INFO:journalctl@ceph.mon.c.smithi188.stdout: num_file_reads_for_auto_readahead: 2 2023-07-20T02:45:35.984 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.write_buffer_size: 33554432 2023-07-20T02:45:35.984 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.max_write_buffer_number: 2 2023-07-20T02:45:35.984 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.compression: NoCompression 2023-07-20T02:45:35.984 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.bottommost_compression: Disabled 2023-07-20T02:45:35.984 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.prefix_extractor: nullptr 2023-07-20T02:45:35.984 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-20T02:45:35.985 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.num_levels: 7 2023-07-20T02:45:35.985 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-20T02:45:35.985 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-20T02:45:35.985 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-20T02:45:35.985 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-20T02:45:35.985 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-20T02:45:35.985 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-20T02:45:35.985 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-20T02:45:35.986 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-20T02:45:35.986 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-20T02:45:35.986 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-20T02:45:35.986 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-20T02:45:35.986 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-20T02:45:35.986 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-20T02:45:35.987 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.compression_opts.level: 32767 2023-07-20T02:45:35.987 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.compression_opts.strategy: 0 2023-07-20T02:45:35.987 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-20T02:45:35.987 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-20T02:45:35.987 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-20T02:45:35.988 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-20T02:45:35.988 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.compression_opts.enabled: false 2023-07-20T02:45:35.988 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-20T02:45:35.988 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-20T02:45:35.988 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-20T02:45:35.988 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-20T02:45:35.988 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.target_file_size_base: 67108864 2023-07-20T02:45:35.989 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-20T02:45:35.989 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-20T02:45:35.989 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-20T02:45:35.989 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-20T02:45:35.989 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-20T02:45:35.989 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-20T02:45:35.989 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-20T02:45:35.990 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-20T02:45:35.990 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-20T02:45:35.990 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-20T02:45:35.990 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-20T02:45:35.990 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-20T02:45:35.990 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-20T02:45:35.990 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-20T02:45:35.990 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.arena_block_size: 1048576 2023-07-20T02:45:35.991 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-20T02:45:35.991 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-20T02:45:35.991 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.disable_auto_compactions: 0 2023-07-20T02:45:35.991 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-20T02:45:35.991 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-20T02:45:35.991 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-20T02:45:35.991 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-20T02:45:35.992 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-20T02:45:35.992 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-20T02:45:35.992 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-20T02:45:35.992 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-20T02:45:35.992 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-20T02:45:35.992 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-20T02:45:35.992 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.table_properties_collectors: 2023-07-20T02:45:35.992 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.inplace_update_support: 0 2023-07-20T02:45:35.993 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-20T02:45:35.993 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-20T02:45:35.993 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-20T02:45:35.993 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-20T02:45:35.993 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.bloom_locality: 0 2023-07-20T02:45:35.993 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.max_successive_merges: 0 2023-07-20T02:45:35.993 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-20T02:45:35.993 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.paranoid_file_checks: 0 2023-07-20T02:45:35.994 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.force_consistency_checks: 1 2023-07-20T02:45:35.994 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.report_bg_io_stats: 0 2023-07-20T02:45:35.994 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.ttl: 2592000 2023-07-20T02:45:35.994 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-20T02:45:35.994 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-20T02:45:35.994 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-20T02:45:35.994 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.enable_blob_files: false 2023-07-20T02:45:35.995 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.min_blob_size: 0 2023-07-20T02:45:35.995 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.blob_file_size: 268435456 2023-07-20T02:45:35.995 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-20T02:45:35.995 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-20T02:45:35.995 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-20T02:45:35.995 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-20T02:45:35.995 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-20T02:45:35.995 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.blob_file_starting_level: 0 2023-07-20T02:45:35.996 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-20T02:45:35.996 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-20T02:45:35.996 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-07-20T02:45:35.996 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ac0aa8eb-7d8e-4e96-9a7f-11bfdfc1822d 2023-07-20T02:45:35.996 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689821135687832, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-07-20T02:45:35.996 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-07-20T02:45:35.996 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689821135688424, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689821135, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ac0aa8eb-7d8e-4e96-9a7f-11bfdfc1822d", "db_session_id": "WL1W0VLCF37XUW4XWB70", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-07-20T02:45:35.997 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689821135688514, "job": 1, "event": "recovery_finished"} 2023-07-20T02:45:35.997 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-07-20T02:45:35.997 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-20T02:45:35.997 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55848fd4c000 2023-07-20T02:45:35.997 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: DB pointer 0x55848fd38000 2023-07-20T02:45:35.997 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-20T02:45:35.997 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-20T02:45:35.998 INFO:journalctl@ceph.mon.c.smithi188.stdout: ** DB Stats ** 2023-07-20T02:45:35.998 INFO:journalctl@ceph.mon.c.smithi188.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T02:45:35.998 INFO:journalctl@ceph.mon.c.smithi188.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-20T02:45:35.998 INFO:journalctl@ceph.mon.c.smithi188.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T02:45:35.998 INFO:journalctl@ceph.mon.c.smithi188.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T02:45:35.998 INFO:journalctl@ceph.mon.c.smithi188.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-20T02:45:35.998 INFO:journalctl@ceph.mon.c.smithi188.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T02:45:35.998 INFO:journalctl@ceph.mon.c.smithi188.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T02:45:35.999 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2023-07-20T02:45:35.999 INFO:journalctl@ceph.mon.c.smithi188.stdout: ** Compaction Stats [default] ** 2023-07-20T02:45:35.999 INFO:journalctl@ceph.mon.c.smithi188.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T02:45:35.999 INFO:journalctl@ceph.mon.c.smithi188.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-20T02:45:35.999 INFO:journalctl@ceph.mon.c.smithi188.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T02:45:35.999 INFO:journalctl@ceph.mon.c.smithi188.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T02:45:35.999 INFO:journalctl@ceph.mon.c.smithi188.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T02:45:35.999 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2023-07-20T02:45:36.000 INFO:journalctl@ceph.mon.c.smithi188.stdout: ** Compaction Stats [default] ** 2023-07-20T02:45:36.000 INFO:journalctl@ceph.mon.c.smithi188.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T02:45:36.000 INFO:journalctl@ceph.mon.c.smithi188.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-20T02:45:36.000 INFO:journalctl@ceph.mon.c.smithi188.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T02:45:36.000 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2023-07-20T02:45:36.000 INFO:journalctl@ceph.mon.c.smithi188.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-20T02:45:36.000 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2023-07-20T02:45:36.001 INFO:journalctl@ceph.mon.c.smithi188.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T02:45:36.001 INFO:journalctl@ceph.mon.c.smithi188.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-20T02:45:36.001 INFO:journalctl@ceph.mon.c.smithi188.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-20T02:45:36.001 INFO:journalctl@ceph.mon.c.smithi188.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-20T02:45:36.001 INFO:journalctl@ceph.mon.c.smithi188.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-20T02:45:36.001 INFO:journalctl@ceph.mon.c.smithi188.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-20T02:45:36.001 INFO:journalctl@ceph.mon.c.smithi188.stdout: Cumulative compaction: 0.00 GB write, 0.45 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T02:45:36.001 INFO:journalctl@ceph.mon.c.smithi188.stdout: Interval compaction: 0.00 GB write, 0.45 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T02:45:36.002 INFO:journalctl@ceph.mon.c.smithi188.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-20T02:45:36.002 INFO:journalctl@ceph.mon.c.smithi188.stdout: Block cache BinnedLRUCache@0x55848fcb3350#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2023-07-20T02:45:36.002 INFO:journalctl@ceph.mon.c.smithi188.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-07-20T02:45:36.002 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2023-07-20T02:45:36.002 INFO:journalctl@ceph.mon.c.smithi188.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-20T02:45:36.002 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: mon.c does not exist in monmap, will attempt to join an existing cluster 2023-07-20T02:45:36.002 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: using public_addr v2:172.21.15.188:0/0 -> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] 2023-07-20T02:45:36.003 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: starting mon.c rank -1 at public addrs [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] at bind addrs [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 1b04643c-26a7-11ee-9b34-001a4aab830c 2023-07-20T02:45:36.003 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: mon.c@-1(???) e0 preinit fsid 1b04643c-26a7-11ee-9b34-001a4aab830c 2023-07-20T02:45:36.003 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 bash[129487]: 2386009301f0c2a645324e851391d657a7c056b56ce2c8043a1194410424ee3e 2023-07-20T02:45:36.003 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: mon.c@-1(synchronizing).mds e1 new map 2023-07-20T02:45:36.003 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: mon.c@-1(synchronizing).mds e1 print_map 2023-07-20T02:45:36.003 INFO:journalctl@ceph.mon.c.smithi188.stdout: e1 2023-07-20T02:45:36.003 INFO:journalctl@ceph.mon.c.smithi188.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-20T02:45:36.003 INFO:journalctl@ceph.mon.c.smithi188.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-20T02:45:36.004 INFO:journalctl@ceph.mon.c.smithi188.stdout: legacy client fscid: -1 2023-07-20T02:45:36.004 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2023-07-20T02:45:36.004 INFO:journalctl@ceph.mon.c.smithi188.stdout: No filesystems configured 2023-07-20T02:45:36.004 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-07-20T02:45:36.004 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-07-20T02:45:36.004 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-07-20T02:45:36.004 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-07-20T02:45:36.005 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-07-20T02:45:36.005 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-07-20T02:45:36.005 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-20T02:45:36.005 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T02:45:36.005 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T02:45:36.005 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T02:45:36.005 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:36.005 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:36.006 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-20T02:45:36.006 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:36.006 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:36.006 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:36.006 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:36.006 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:36.006 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi158", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:45:36.006 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:45:36.007 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:45:36.007 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: Updating smithi158:/etc/ceph/ceph.conf 2023-07-20T02:45:36.007 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: Updating smithi158:/var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T02:45:36.007 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:36.007 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:36.007 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:36.007 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:36.007 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: Updating smithi158:/etc/ceph/ceph.client.admin.keyring 2023-07-20T02:45:36.008 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: Updating smithi158:/var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T02:45:36.008 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi188", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:45:36.008 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: Deploying cephadm binary to smithi188 2023-07-20T02:45:36.008 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:36.008 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:36.008 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: Added host smithi188 2023-07-20T02:45:36.008 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:45:36.009 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:36.009 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:36.009 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:36.009 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:36.009 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-20T02:45:36.009 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:36.009 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:36.009 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:36.010 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:36.010 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:36.010 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:36.010 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi188", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:45:36.010 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:45:36.010 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:45:36.010 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: Updating smithi188:/etc/ceph/ceph.conf 2023-07-20T02:45:36.010 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: Updating smithi188:/var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T02:45:36.011 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2023-07-20T02:45:36.011 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: Updating smithi188:/var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T02:45:36.011 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/2606875359' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-20T02:45:36.011 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:36.011 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:36.011 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:36.011 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/2606875359' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-20T02:45:36.011 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T02:45:36.012 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:36.012 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi007:172.21.15.7=a;smithi158:172.21.15.158=b;smithi188:172.21.15.188=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:45:36.012 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: Saving service mon spec with placement smithi007:172.21.15.7=a;smithi158:172.21.15.158=b;smithi188:172.21.15.188=c;count:3 2023-07-20T02:45:36.012 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:36.012 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:45:36.012 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:45:36.012 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:45:36.013 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:36.013 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T02:45:36.013 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:45:36.013 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: Deploying daemon mon.c on smithi188 2023-07-20T02:45:36.013 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:36.013 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:36.013 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: from='client.? 172.21.15.188:0/963633529' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T02:45:36.013 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-07-20T02:45:36.014 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 ceph-mon[129553]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-07-20T02:45:36.014 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:35 smithi188 systemd[1]: Started Ceph mon.c for 1b04643c-26a7-11ee-9b34-001a4aab830c. 2023-07-20T02:45:36.554 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-20T02:45:36.554 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph mon dump -f json 2023-07-20T02:45:37.122 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T02:45:41.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:40 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:45:41.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:40 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:45:41.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:40 smithi007 ceph-mon[126353]: mon.a calling monitor election 2023-07-20T02:45:41.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:40 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:45:41.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:40 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:45:41.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:40 smithi007 ceph-mon[126353]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:41.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:40 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:45:41.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:40 smithi007 ceph-mon[126353]: mon.c calling monitor election 2023-07-20T02:45:41.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:40 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:45:41.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:40 smithi007 ceph-mon[126353]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:41.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:40 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:45:41.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:40 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:45:41.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:40 smithi007 ceph-mon[126353]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T02:45:41.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:40 smithi007 ceph-mon[126353]: monmap e2: 2 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],c=[v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0]} removed_ranks: {} 2023-07-20T02:45:41.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:40 smithi007 ceph-mon[126353]: fsmap 2023-07-20T02:45:41.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:40 smithi007 ceph-mon[126353]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T02:45:41.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:40 smithi007 ceph-mon[126353]: mgrmap e13: x(active, since 47s) 2023-07-20T02:45:41.198 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:40 smithi007 ceph-mon[126353]: overall HEALTH_OK 2023-07-20T02:45:41.198 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:40 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:41.198 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:40 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:41.209 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:40 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:45:41.209 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:40 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:45:41.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:40 smithi188 ceph-mon[129553]: mon.a calling monitor election 2023-07-20T02:45:41.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:40 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:45:41.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:40 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:45:41.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:40 smithi188 ceph-mon[129553]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:41.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:40 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:45:41.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:40 smithi188 ceph-mon[129553]: mon.c calling monitor election 2023-07-20T02:45:41.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:40 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:45:41.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:40 smithi188 ceph-mon[129553]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:41.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:40 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:45:41.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:40 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:45:41.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:40 smithi188 ceph-mon[129553]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T02:45:41.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:40 smithi188 ceph-mon[129553]: monmap e2: 2 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],c=[v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0]} removed_ranks: {} 2023-07-20T02:45:41.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:40 smithi188 ceph-mon[129553]: fsmap 2023-07-20T02:45:41.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:40 smithi188 ceph-mon[129553]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T02:45:41.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:40 smithi188 ceph-mon[129553]: mgrmap e13: x(active, since 47s) 2023-07-20T02:45:41.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:40 smithi188 ceph-mon[129553]: overall HEALTH_OK 2023-07-20T02:45:41.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:40 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:41.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:40 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:41.513 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.504+0000 7fbf00490700 1 -- 172.21.15.188:0/1412439916 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbee40035c0 msgr2=0x7fbee4005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:41.514 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.504+0000 7fbf00490700 1 --2- 172.21.15.188:0/1412439916 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbee40035c0 0x7fbee4005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbef80ff170 tx=0x7fbef400feb0 comp rx=0 tx=0).stop 2023-07-20T02:45:41.514 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.504+0000 7fbf00490700 1 -- 172.21.15.188:0/1412439916 shutdown_connections 2023-07-20T02:45:41.514 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.504+0000 7fbf00490700 1 --2- 172.21.15.188:0/1412439916 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbee40035c0 0x7fbee4005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:41.514 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.504+0000 7fbf00490700 1 -- 172.21.15.188:0/1412439916 >> 172.21.15.188:0/1412439916 conn(0x7fbef80fad70 msgr2=0x7fbef80fd190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:45:41.514 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.504+0000 7fbf00490700 1 -- 172.21.15.188:0/1412439916 shutdown_connections 2023-07-20T02:45:41.514 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.504+0000 7fbf00490700 1 -- 172.21.15.188:0/1412439916 wait complete. 2023-07-20T02:45:41.515 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.505+0000 7fbf00490700 1 Processor -- start 2023-07-20T02:45:41.516 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.505+0000 7fbf00490700 1 -- start start 2023-07-20T02:45:41.516 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.506+0000 7fbf00490700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbee40035c0 0x7fbef8196710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:41.516 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.506+0000 7fbf00490700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbef8196c50 0x7fbef81970b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:41.516 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.506+0000 7fbf00490700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef819f780 con 0x7fbee40035c0 2023-07-20T02:45:41.516 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.506+0000 7fbf00490700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef819f8c0 con 0x7fbef8196c50 2023-07-20T02:45:41.516 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.506+0000 7fbefda2b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbef8196c50 0x7fbef81970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:41.516 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.506+0000 7fbefda2b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbef8196c50 0x7fbef81970b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:34782/0 (socket says 172.21.15.188:34782) 2023-07-20T02:45:41.517 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.506+0000 7fbefda2b700 1 -- 172.21.15.188:0/3314618898 learned_addr learned my addr 172.21.15.188:0/3314618898 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-07-20T02:45:41.517 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.506+0000 7fbefda2b700 1 -- 172.21.15.188:0/3314618898 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbef8196c50 msgr2=0x7fbef81970b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-07-20T02:45:41.517 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.506+0000 7fbefda2b700 1 -- 172.21.15.188:0/3314618898 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbef8196c50 msgr2=0x7fbef81970b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-20T02:45:41.517 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.506+0000 7fbefda2b700 1 --2- 172.21.15.188:0/3314618898 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbef8196c50 0x7fbef81970b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-20T02:45:41.517 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.506+0000 7fbefe22c700 1 --2- 172.21.15.188:0/3314618898 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbee40035c0 0x7fbef8196710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:41.517 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.506+0000 7fbefda2b700 1 --2- 172.21.15.188:0/3314618898 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbef8196c50 0x7fbef81970b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T02:45:41.517 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.507+0000 7fbefe22c700 1 -- 172.21.15.188:0/3314618898 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbef8196c50 msgr2=0x7fbef81970b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T02:45:41.517 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.507+0000 7fbefe22c700 1 --2- 172.21.15.188:0/3314618898 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbef8196c50 0x7fbef81970b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:41.518 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.507+0000 7fbefe22c700 1 -- 172.21.15.188:0/3314618898 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef400f900 con 0x7fbee40035c0 2023-07-20T02:45:41.519 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.507+0000 7fbefe22c700 1 --2- 172.21.15.188:0/3314618898 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbee40035c0 0x7fbef8196710 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbef80ff170 tx=0x7fbef40047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:45:41.519 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.507+0000 7fbeef7fe700 1 -- 172.21.15.188:0/3314618898 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fbef4013b00 con 0x7fbee40035c0 2023-07-20T02:45:41.519 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.507+0000 7fbeef7fe700 1 -- 172.21.15.188:0/3314618898 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbef40044f0 con 0x7fbee40035c0 2023-07-20T02:45:41.519 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.507+0000 7fbf00490700 1 -- 172.21.15.188:0/3314618898 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbef819fac0 con 0x7fbee40035c0 2023-07-20T02:45:41.519 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.507+0000 7fbeef7fe700 1 -- 172.21.15.188:0/3314618898 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fbef4003be0 con 0x7fbee40035c0 2023-07-20T02:45:41.519 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.507+0000 7fbf00490700 1 -- 172.21.15.188:0/3314618898 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbef81a0010 con 0x7fbee40035c0 2023-07-20T02:45:41.520 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.508+0000 7fbeef7fe700 1 -- 172.21.15.188:0/3314618898 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7fbef4004030 con 0x7fbee40035c0 2023-07-20T02:45:41.520 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.509+0000 7fbeef7fe700 1 --2- 172.21.15.188:0/3314618898 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fbedc038630 0x7fbedc03aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:41.520 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.509+0000 7fbf00490700 1 -- 172.21.15.188:0/3314618898 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbef8061da0 con 0x7fbee40035c0 2023-07-20T02:45:41.520 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.509+0000 7fbeef7fe700 1 -- 172.21.15.188:0/3314618898 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fbef404dca0 con 0x7fbee40035c0 2023-07-20T02:45:41.520 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.509+0000 7fbefda2b700 1 --2- 172.21.15.188:0/3314618898 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fbedc038630 0x7fbedc03aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:41.520 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.510+0000 7fbefda2b700 1 --2- 172.21.15.188:0/3314618898 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fbedc038630 0x7fbedc03aaf0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbee8009180 tx=0x7fbee8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:45:41.523 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.514+0000 7fbeef7fe700 1 -- 172.21.15.188:0/3314618898 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fbef404de60 con 0x7fbee40035c0 2023-07-20T02:45:41.791 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.782+0000 7fbf00490700 1 -- 172.21.15.188:0/3314618898 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbef81a0410 con 0x7fbee40035c0 2023-07-20T02:45:41.792 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.784+0000 7fbeef7fe700 1 -- 172.21.15.188:0/3314618898 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1002 (secure 0 0 0) 0x7fbef4011540 con 0x7fbee40035c0 2023-07-20T02:45:41.792 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-07-20T02:45:41.792 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":2,"fsid":"1b04643c-26a7-11ee-9b34-001a4aab830c","modified":"2023-07-20T02:45:35.720398Z","created":"2023-07-20T02:43:46.393290Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3300","nonce":0},{"type":"v1","addr":"172.21.15.7:6789","nonce":0}]},"addr":"172.21.15.7:6789/0","public_addr":"172.21.15.7:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:3300","nonce":0},{"type":"v1","addr":"172.21.15.188:6789","nonce":0}]},"addr":"172.21.15.188:6789/0","public_addr":"172.21.15.188:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-07-20T02:45:41.794 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.785+0000 7fbf00490700 1 -- 172.21.15.188:0/3314618898 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fbedc038630 msgr2=0x7fbedc03aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:41.794 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.785+0000 7fbf00490700 1 --2- 172.21.15.188:0/3314618898 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fbedc038630 0x7fbedc03aaf0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbee8009180 tx=0x7fbee8008040 comp rx=0 tx=0).stop 2023-07-20T02:45:41.794 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.785+0000 7fbf00490700 1 -- 172.21.15.188:0/3314618898 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbee40035c0 msgr2=0x7fbef8196710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:41.794 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.785+0000 7fbf00490700 1 --2- 172.21.15.188:0/3314618898 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbee40035c0 0x7fbef8196710 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbef80ff170 tx=0x7fbef40047a0 comp rx=0 tx=0).stop 2023-07-20T02:45:41.795 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.785+0000 7fbf00490700 1 -- 172.21.15.188:0/3314618898 shutdown_connections 2023-07-20T02:45:41.795 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.785+0000 7fbf00490700 1 --2- 172.21.15.188:0/3314618898 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fbedc038630 0x7fbedc03aaf0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:41.795 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.785+0000 7fbf00490700 1 --2- 172.21.15.188:0/3314618898 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbee40035c0 0x7fbef8196710 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:41.795 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.785+0000 7fbf00490700 1 --2- 172.21.15.188:0/3314618898 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbef8196c50 0x7fbef81970b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:41.795 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.785+0000 7fbf00490700 1 -- 172.21.15.188:0/3314618898 >> 172.21.15.188:0/3314618898 conn(0x7fbef80fad70 msgr2=0x7fbef80fc2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:45:41.795 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.785+0000 7fbf00490700 1 -- 172.21.15.188:0/3314618898 shutdown_connections 2023-07-20T02:45:41.795 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:41.785+0000 7fbf00490700 1 -- 172.21.15.188:0/3314618898 wait complete. 2023-07-20T02:45:41.796 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 2 2023-07-20T02:45:42.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:41 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:42.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:41 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T02:45:42.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:41 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:45:42.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:41 smithi188 ceph-mon[129553]: Deploying daemon mon.b on smithi158 2023-07-20T02:45:42.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:41 smithi188 ceph-mon[129553]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:42.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:41 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:45:42.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:41 smithi188 ceph-mon[129553]: from='client.? 172.21.15.188:0/3314618898' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T02:45:42.193 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:41 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:42.193 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:41 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T02:45:42.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:41 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:45:42.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:41 smithi007 ceph-mon[126353]: Deploying daemon mon.b on smithi158 2023-07-20T02:45:42.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:41 smithi007 ceph-mon[126353]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:42.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:41 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:45:42.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:41 smithi007 ceph-mon[126353]: from='client.? 172.21.15.188:0/3314618898' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T02:45:43.185 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-20T02:45:43.186 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph mon dump -f json 2023-07-20T02:45:43.193 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:45:42 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:45:42.720+0000 7f610e4dd700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-07-20T02:45:43.504 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 systemd[1]: Starting Ceph mon.b for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:45:43.750 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T02:45:43.860 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 podman[128885]: 2023-07-20T02:45:44.136 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 bash[128885]: aaade1924673e86a4fbe08b79e845bb1241e68c4715ce44febb588b1899a07c8 2023-07-20T02:45:44.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: set uid:gid to 167:167 (ceph:ceph) 2023-07-20T02:45:44.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev), process ceph-mon, pid 7 2023-07-20T02:45:44.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: pidfile_write: ignore empty --pid-file 2023-07-20T02:45:44.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: load: jerasure load: lrc 2023-07-20T02:45:44.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: RocksDB version: 7.9.2 2023-07-20T02:45:44.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Git sha 0 2023-07-20T02:45:44.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Compile date 2023-07-19 21:36:35 2023-07-20T02:45:44.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: DB SUMMARY 2023-07-20T02:45:44.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: DB Session ID: 7J3KQ0COVN2K2FPID8AH 2023-07-20T02:45:44.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: CURRENT file: CURRENT 2023-07-20T02:45:44.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: IDENTITY file: IDENTITY 2023-07-20T02:45:44.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-07-20T02:45:44.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2023-07-20T02:45:44.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2023-07-20T02:45:44.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.error_if_exists: 0 2023-07-20T02:45:44.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.create_if_missing: 0 2023-07-20T02:45:44.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.paranoid_checks: 1 2023-07-20T02:45:44.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-20T02:45:44.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-20T02:45:44.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-20T02:45:44.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.env: 0x55c8c5bf8c60 2023-07-20T02:45:44.140 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.fs: PosixFileSystem 2023-07-20T02:45:44.140 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.info_log: 0x55c8c811d2c0 2023-07-20T02:45:44.140 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.max_file_opening_threads: 16 2023-07-20T02:45:44.140 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.statistics: (nil) 2023-07-20T02:45:44.140 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.use_fsync: 0 2023-07-20T02:45:44.140 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.max_log_file_size: 0 2023-07-20T02:45:44.140 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-20T02:45:44.141 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-20T02:45:44.141 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.keep_log_file_num: 1000 2023-07-20T02:45:44.141 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.recycle_log_file_num: 0 2023-07-20T02:45:44.141 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.allow_fallocate: 1 2023-07-20T02:45:44.141 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.allow_mmap_reads: 0 2023-07-20T02:45:44.141 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.allow_mmap_writes: 0 2023-07-20T02:45:44.142 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.use_direct_reads: 0 2023-07-20T02:45:44.142 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-20T02:45:44.142 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.create_missing_column_families: 0 2023-07-20T02:45:44.142 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.db_log_dir: 2023-07-20T02:45:44.142 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.wal_dir: 2023-07-20T02:45:44.142 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-20T02:45:44.142 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-20T02:45:44.143 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-20T02:45:44.143 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-20T02:45:44.143 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-20T02:45:44.143 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-20T02:45:44.143 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.advise_random_on_open: 1 2023-07-20T02:45:44.143 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.db_write_buffer_size: 0 2023-07-20T02:45:44.143 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.write_buffer_manager: 0x55c8c803e3c0 2023-07-20T02:45:44.143 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-20T02:45:44.144 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-20T02:45:44.144 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-20T02:45:44.144 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.rate_limiter: (nil) 2023-07-20T02:45:44.144 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-20T02:45:44.144 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.wal_recovery_mode: 2 2023-07-20T02:45:44.144 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.enable_thread_tracking: 0 2023-07-20T02:45:44.144 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.enable_pipelined_write: 0 2023-07-20T02:45:44.145 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.unordered_write: 0 2023-07-20T02:45:44.145 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-20T02:45:44.145 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-20T02:45:44.145 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-20T02:45:44.145 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-20T02:45:44.145 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.row_cache: None 2023-07-20T02:45:44.145 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.wal_filter: None 2023-07-20T02:45:44.146 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-20T02:45:44.146 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.allow_ingest_behind: 0 2023-07-20T02:45:44.146 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.two_write_queues: 0 2023-07-20T02:45:44.146 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.manual_wal_flush: 0 2023-07-20T02:45:44.146 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.wal_compression: 0 2023-07-20T02:45:44.146 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.atomic_flush: 0 2023-07-20T02:45:44.146 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-20T02:45:44.146 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-20T02:45:44.147 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-20T02:45:44.147 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.log_readahead_size: 0 2023-07-20T02:45:44.147 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-20T02:45:44.147 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.best_efforts_recovery: 0 2023-07-20T02:45:44.147 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-20T02:45:44.147 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-20T02:45:44.147 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.allow_data_in_errors: 0 2023-07-20T02:45:44.149 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.db_host_id: __hostname__ 2023-07-20T02:45:44.149 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-20T02:45:44.149 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.max_background_jobs: 2 2023-07-20T02:45:44.149 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.max_background_compactions: -1 2023-07-20T02:45:44.149 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.max_subcompactions: 1 2023-07-20T02:45:44.149 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-20T02:45:44.150 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-20T02:45:44.150 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-20T02:45:44.150 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.max_total_wal_size: 0 2023-07-20T02:45:44.150 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-20T02:45:44.150 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-20T02:45:44.150 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-20T02:45:44.150 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-20T02:45:44.151 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.max_open_files: -1 2023-07-20T02:45:44.151 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.bytes_per_sync: 0 2023-07-20T02:45:44.151 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-20T02:45:44.151 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-20T02:45:44.151 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.compaction_readahead_size: 0 2023-07-20T02:45:44.151 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.max_background_flushes: -1 2023-07-20T02:45:44.151 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Compression algorithms supported: 2023-07-20T02:45:44.152 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-20T02:45:44.152 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: kZSTD supported: 0 2023-07-20T02:45:44.152 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: kXpressCompression supported: 0 2023-07-20T02:45:44.152 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: kLZ4HCCompression supported: 1 2023-07-20T02:45:44.152 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: kZlibCompression supported: 1 2023-07-20T02:45:44.152 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: kSnappyCompression supported: 1 2023-07-20T02:45:44.152 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: kLZ4Compression supported: 1 2023-07-20T02:45:44.152 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: kBZip2Compression supported: 0 2023-07-20T02:45:44.153 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-20T02:45:44.153 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-20T02:45:44.153 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2023-07-20T02:45:44.153 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-20T02:45:44.153 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-20T02:45:44.153 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.merge_operator: 2023-07-20T02:45:44.153 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.compaction_filter: None 2023-07-20T02:45:44.153 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.compaction_filter_factory: None 2023-07-20T02:45:44.154 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.sst_partitioner_factory: None 2023-07-20T02:45:44.154 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-20T02:45:44.154 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-20T02:45:44.154 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c8c811d3c0) 2023-07-20T02:45:44.154 INFO:journalctl@ceph.mon.b.smithi158.stdout: cache_index_and_filter_blocks: 1 2023-07-20T02:45:44.154 INFO:journalctl@ceph.mon.b.smithi158.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-20T02:45:44.154 INFO:journalctl@ceph.mon.b.smithi158.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-20T02:45:44.155 INFO:journalctl@ceph.mon.b.smithi158.stdout: pin_top_level_index_and_filter: 1 2023-07-20T02:45:44.155 INFO:journalctl@ceph.mon.b.smithi158.stdout: index_type: 0 2023-07-20T02:45:44.155 INFO:journalctl@ceph.mon.b.smithi158.stdout: data_block_index_type: 0 2023-07-20T02:45:44.155 INFO:journalctl@ceph.mon.b.smithi158.stdout: index_shortening: 1 2023-07-20T02:45:44.155 INFO:journalctl@ceph.mon.b.smithi158.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-20T02:45:44.155 INFO:journalctl@ceph.mon.b.smithi158.stdout: checksum: 4 2023-07-20T02:45:44.155 INFO:journalctl@ceph.mon.b.smithi158.stdout: no_block_cache: 0 2023-07-20T02:45:44.155 INFO:journalctl@ceph.mon.b.smithi158.stdout: block_cache: 0x55c8c80dd1f0 2023-07-20T02:45:44.156 INFO:journalctl@ceph.mon.b.smithi158.stdout: block_cache_name: BinnedLRUCache 2023-07-20T02:45:44.156 INFO:journalctl@ceph.mon.b.smithi158.stdout: block_cache_options: 2023-07-20T02:45:44.156 INFO:journalctl@ceph.mon.b.smithi158.stdout: capacity : 536870912 2023-07-20T02:45:44.156 INFO:journalctl@ceph.mon.b.smithi158.stdout: num_shard_bits : 4 2023-07-20T02:45:44.156 INFO:journalctl@ceph.mon.b.smithi158.stdout: strict_capacity_limit : 0 2023-07-20T02:45:44.156 INFO:journalctl@ceph.mon.b.smithi158.stdout: high_pri_pool_ratio: 0.000 2023-07-20T02:45:44.156 INFO:journalctl@ceph.mon.b.smithi158.stdout: block_cache_compressed: (nil) 2023-07-20T02:45:44.156 INFO:journalctl@ceph.mon.b.smithi158.stdout: persistent_cache: (nil) 2023-07-20T02:45:44.157 INFO:journalctl@ceph.mon.b.smithi158.stdout: block_size: 4096 2023-07-20T02:45:44.157 INFO:journalctl@ceph.mon.b.smithi158.stdout: block_size_deviation: 10 2023-07-20T02:45:44.157 INFO:journalctl@ceph.mon.b.smithi158.stdout: block_restart_interval: 16 2023-07-20T02:45:44.157 INFO:journalctl@ceph.mon.b.smithi158.stdout: index_block_restart_interval: 1 2023-07-20T02:45:44.157 INFO:journalctl@ceph.mon.b.smithi158.stdout: metadata_block_size: 4096 2023-07-20T02:45:44.157 INFO:journalctl@ceph.mon.b.smithi158.stdout: partition_filters: 0 2023-07-20T02:45:44.157 INFO:journalctl@ceph.mon.b.smithi158.stdout: use_delta_encoding: 1 2023-07-20T02:45:44.157 INFO:journalctl@ceph.mon.b.smithi158.stdout: filter_policy: bloomfilter 2023-07-20T02:45:44.158 INFO:journalctl@ceph.mon.b.smithi158.stdout: whole_key_filtering: 1 2023-07-20T02:45:44.158 INFO:journalctl@ceph.mon.b.smithi158.stdout: verify_compression: 0 2023-07-20T02:45:44.158 INFO:journalctl@ceph.mon.b.smithi158.stdout: read_amp_bytes_per_bit: 0 2023-07-20T02:45:44.158 INFO:journalctl@ceph.mon.b.smithi158.stdout: format_version: 5 2023-07-20T02:45:44.158 INFO:journalctl@ceph.mon.b.smithi158.stdout: enable_index_compression: 1 2023-07-20T02:45:44.158 INFO:journalctl@ceph.mon.b.smithi158.stdout: block_align: 0 2023-07-20T02:45:44.158 INFO:journalctl@ceph.mon.b.smithi158.stdout: max_auto_readahead_size: 262144 2023-07-20T02:45:44.158 INFO:journalctl@ceph.mon.b.smithi158.stdout: prepopulate_block_cache: 0 2023-07-20T02:45:44.159 INFO:journalctl@ceph.mon.b.smithi158.stdout: initial_auto_readahead_size: 8192 2023-07-20T02:45:44.159 INFO:journalctl@ceph.mon.b.smithi158.stdout: num_file_reads_for_auto_readahead: 2 2023-07-20T02:45:44.159 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.write_buffer_size: 33554432 2023-07-20T02:45:44.159 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.max_write_buffer_number: 2 2023-07-20T02:45:44.159 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.compression: NoCompression 2023-07-20T02:45:44.159 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.bottommost_compression: Disabled 2023-07-20T02:45:44.159 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.prefix_extractor: nullptr 2023-07-20T02:45:44.159 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-20T02:45:44.160 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.num_levels: 7 2023-07-20T02:45:44.160 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-20T02:45:44.160 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-20T02:45:44.160 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-20T02:45:44.160 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-20T02:45:44.160 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-20T02:45:44.161 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-20T02:45:44.161 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-20T02:45:44.161 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-20T02:45:44.161 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-20T02:45:44.161 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-20T02:45:44.161 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-20T02:45:44.161 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-20T02:45:44.162 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-20T02:45:44.162 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.compression_opts.level: 32767 2023-07-20T02:45:44.162 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.compression_opts.strategy: 0 2023-07-20T02:45:44.162 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-20T02:45:44.162 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-20T02:45:44.162 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-20T02:45:44.162 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-20T02:45:44.162 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.compression_opts.enabled: false 2023-07-20T02:45:44.163 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-20T02:45:44.163 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-20T02:45:44.163 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-20T02:45:44.163 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-20T02:45:44.163 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.target_file_size_base: 67108864 2023-07-20T02:45:44.163 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-20T02:45:44.163 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-20T02:45:44.163 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-20T02:45:44.164 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-20T02:45:44.164 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-20T02:45:44.164 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-20T02:45:44.164 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-20T02:45:44.164 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-20T02:45:44.164 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-20T02:45:44.164 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-20T02:45:44.164 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-20T02:45:44.165 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-20T02:45:44.165 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-20T02:45:44.165 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-20T02:45:44.165 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.arena_block_size: 1048576 2023-07-20T02:45:44.165 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-20T02:45:44.165 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-20T02:45:44.165 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.disable_auto_compactions: 0 2023-07-20T02:45:44.166 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-20T02:45:44.166 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-20T02:45:44.166 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-20T02:45:44.166 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-20T02:45:44.166 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-20T02:45:44.166 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-20T02:45:44.166 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-20T02:45:44.166 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-20T02:45:44.167 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-20T02:45:44.167 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-20T02:45:44.167 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.table_properties_collectors: 2023-07-20T02:45:44.167 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.inplace_update_support: 0 2023-07-20T02:45:44.167 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-20T02:45:44.167 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-20T02:45:44.167 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-20T02:45:44.168 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-20T02:45:44.168 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.bloom_locality: 0 2023-07-20T02:45:44.168 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.max_successive_merges: 0 2023-07-20T02:45:44.168 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-20T02:45:44.168 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.paranoid_file_checks: 0 2023-07-20T02:45:44.168 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.force_consistency_checks: 1 2023-07-20T02:45:44.168 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.report_bg_io_stats: 0 2023-07-20T02:45:44.168 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.ttl: 2592000 2023-07-20T02:45:44.169 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-20T02:45:44.169 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-20T02:45:44.169 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-20T02:45:44.169 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.enable_blob_files: false 2023-07-20T02:45:44.169 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.min_blob_size: 0 2023-07-20T02:45:44.169 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.blob_file_size: 268435456 2023-07-20T02:45:44.169 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-20T02:45:44.169 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-20T02:45:44.170 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-20T02:45:44.170 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-20T02:45:44.170 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-20T02:45:44.170 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.blob_file_starting_level: 0 2023-07-20T02:45:44.170 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-20T02:45:44.170 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-20T02:45:44.170 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-07-20T02:45:44.171 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 7f336721-299b-40a2-aec8-74f58a2842c8 2023-07-20T02:45:44.171 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689821143858586, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-07-20T02:45:44.171 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-07-20T02:45:44.171 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689821143858997, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689821143, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7f336721-299b-40a2-aec8-74f58a2842c8", "db_session_id": "7J3KQ0COVN2K2FPID8AH", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-07-20T02:45:44.171 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689821143859064, "job": 1, "event": "recovery_finished"} 2023-07-20T02:45:44.171 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-07-20T02:45:44.171 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-20T02:45:44.172 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c8c8176000 2023-07-20T02:45:44.172 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: DB pointer 0x55c8c8162000 2023-07-20T02:45:44.172 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-20T02:45:44.172 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-20T02:45:44.172 INFO:journalctl@ceph.mon.b.smithi158.stdout: ** DB Stats ** 2023-07-20T02:45:44.172 INFO:journalctl@ceph.mon.b.smithi158.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T02:45:44.172 INFO:journalctl@ceph.mon.b.smithi158.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-20T02:45:44.172 INFO:journalctl@ceph.mon.b.smithi158.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T02:45:44.173 INFO:journalctl@ceph.mon.b.smithi158.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T02:45:44.173 INFO:journalctl@ceph.mon.b.smithi158.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-20T02:45:44.173 INFO:journalctl@ceph.mon.b.smithi158.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T02:45:44.173 INFO:journalctl@ceph.mon.b.smithi158.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T02:45:44.173 INFO:journalctl@ceph.mon.b.smithi158.stdout: 2023-07-20T02:45:44.173 INFO:journalctl@ceph.mon.b.smithi158.stdout: ** Compaction Stats [default] ** 2023-07-20T02:45:44.173 INFO:journalctl@ceph.mon.b.smithi158.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T02:45:44.174 INFO:journalctl@ceph.mon.b.smithi158.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-20T02:45:44.174 INFO:journalctl@ceph.mon.b.smithi158.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-20T02:45:44.174 INFO:journalctl@ceph.mon.b.smithi158.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-20T02:45:44.174 INFO:journalctl@ceph.mon.b.smithi158.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-20T02:45:44.174 INFO:journalctl@ceph.mon.b.smithi158.stdout: 2023-07-20T02:45:44.174 INFO:journalctl@ceph.mon.b.smithi158.stdout: ** Compaction Stats [default] ** 2023-07-20T02:45:44.174 INFO:journalctl@ceph.mon.b.smithi158.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T02:45:44.174 INFO:journalctl@ceph.mon.b.smithi158.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-20T02:45:44.175 INFO:journalctl@ceph.mon.b.smithi158.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-20T02:45:44.175 INFO:journalctl@ceph.mon.b.smithi158.stdout: 2023-07-20T02:45:44.175 INFO:journalctl@ceph.mon.b.smithi158.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-20T02:45:44.175 INFO:journalctl@ceph.mon.b.smithi158.stdout: 2023-07-20T02:45:44.175 INFO:journalctl@ceph.mon.b.smithi158.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T02:45:44.175 INFO:journalctl@ceph.mon.b.smithi158.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-20T02:45:44.175 INFO:journalctl@ceph.mon.b.smithi158.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-20T02:45:44.175 INFO:journalctl@ceph.mon.b.smithi158.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-20T02:45:44.176 INFO:journalctl@ceph.mon.b.smithi158.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-20T02:45:44.176 INFO:journalctl@ceph.mon.b.smithi158.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-20T02:45:44.176 INFO:journalctl@ceph.mon.b.smithi158.stdout: Cumulative compaction: 0.00 GB write, 0.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T02:45:44.176 INFO:journalctl@ceph.mon.b.smithi158.stdout: Interval compaction: 0.00 GB write, 0.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T02:45:44.176 INFO:journalctl@ceph.mon.b.smithi158.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-20T02:45:44.176 INFO:journalctl@ceph.mon.b.smithi158.stdout: Block cache BinnedLRUCache@0x55c8c80dd1f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2023-07-20T02:45:44.177 INFO:journalctl@ceph.mon.b.smithi158.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-07-20T02:45:44.177 INFO:journalctl@ceph.mon.b.smithi158.stdout: 2023-07-20T02:45:44.177 INFO:journalctl@ceph.mon.b.smithi158.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-20T02:45:44.177 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: mon.b does not exist in monmap, will attempt to join an existing cluster 2023-07-20T02:45:44.177 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: using public_addr v2:172.21.15.158:0/0 -> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] 2023-07-20T02:45:44.177 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: starting mon.b rank -1 at public addrs [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] at bind addrs [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 1b04643c-26a7-11ee-9b34-001a4aab830c 2023-07-20T02:45:44.177 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: mon.b@-1(???) e0 preinit fsid 1b04643c-26a7-11ee-9b34-001a4aab830c 2023-07-20T02:45:44.178 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: mon.b@-1(synchronizing).mds e1 new map 2023-07-20T02:45:44.178 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: mon.b@-1(synchronizing).mds e1 print_map 2023-07-20T02:45:44.178 INFO:journalctl@ceph.mon.b.smithi158.stdout: e1 2023-07-20T02:45:44.178 INFO:journalctl@ceph.mon.b.smithi158.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-20T02:45:44.178 INFO:journalctl@ceph.mon.b.smithi158.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-20T02:45:44.178 INFO:journalctl@ceph.mon.b.smithi158.stdout: legacy client fscid: -1 2023-07-20T02:45:44.178 INFO:journalctl@ceph.mon.b.smithi158.stdout: 2023-07-20T02:45:44.178 INFO:journalctl@ceph.mon.b.smithi158.stdout: No filesystems configured 2023-07-20T02:45:44.179 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-07-20T02:45:44.179 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-07-20T02:45:44.179 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-07-20T02:45:44.179 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-07-20T02:45:44.179 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-07-20T02:45:44.179 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-07-20T02:45:44.179 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-20T02:45:44.179 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T02:45:44.180 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T02:45:44.180 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T02:45:44.180 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:44.180 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:44.180 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-20T02:45:44.180 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:44.180 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:44.180 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:44.181 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:44.181 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:44.181 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi158", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:45:44.181 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:45:44.181 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:45:44.181 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: Updating smithi158:/etc/ceph/ceph.conf 2023-07-20T02:45:44.181 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: Updating smithi158:/var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T02:45:44.182 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:44.182 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:44.182 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:44.182 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:44.182 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: Updating smithi158:/etc/ceph/ceph.client.admin.keyring 2023-07-20T02:45:44.182 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: Updating smithi158:/var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T02:45:44.182 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi188", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:45:44.183 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: Deploying cephadm binary to smithi188 2023-07-20T02:45:44.183 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:44.183 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:44.183 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: Added host smithi188 2023-07-20T02:45:44.183 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:45:44.183 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:44.183 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:44.183 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:44.184 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:44.184 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-20T02:45:44.184 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:44.184 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:44.184 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:44.184 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:44.184 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:44.184 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:44.185 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi188", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:45:44.185 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:45:44.185 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:45:44.185 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: Updating smithi188:/etc/ceph/ceph.conf 2023-07-20T02:45:44.185 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: Updating smithi188:/var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T02:45:44.185 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2023-07-20T02:45:44.186 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: Updating smithi188:/var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T02:45:44.186 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/2606875359' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-20T02:45:44.186 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:44.186 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:44.186 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:44.186 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/2606875359' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-20T02:45:44.186 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T02:45:44.187 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:44.187 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi007:172.21.15.7=a;smithi158:172.21.15.158=b;smithi188:172.21.15.188=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:45:44.187 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: Saving service mon spec with placement smithi007:172.21.15.7=a;smithi158:172.21.15.158=b;smithi188:172.21.15.188=c;count:3 2023-07-20T02:45:44.187 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:44.187 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:45:44.187 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:45:44.187 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:45:44.188 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:44.188 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T02:45:44.188 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:45:44.188 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: Deploying daemon mon.c on smithi188 2023-07-20T02:45:44.188 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:44.188 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:44.189 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='client.? 172.21.15.188:0/963633529' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T02:45:44.189 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:45:44.189 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:45:44.189 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: mon.a calling monitor election 2023-07-20T02:45:44.189 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:45:44.189 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:45:44.189 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:44.189 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:45:44.190 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: mon.c calling monitor election 2023-07-20T02:45:44.190 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:45:44.190 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:44.190 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:45:44.190 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:45:44.190 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T02:45:44.190 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: monmap e2: 2 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],c=[v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0]} removed_ranks: {} 2023-07-20T02:45:44.191 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: fsmap 2023-07-20T02:45:44.191 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T02:45:44.191 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: mgrmap e13: x(active, since 47s) 2023-07-20T02:45:44.191 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: overall HEALTH_OK 2023-07-20T02:45:44.191 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:44.191 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:44.191 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:44.191 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T02:45:44.192 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:45:44.192 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: Deploying daemon mon.b on smithi158 2023-07-20T02:45:44.192 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:44.192 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:45:44.192 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: from='client.? 172.21.15.188:0/3314618898' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T02:45:44.192 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-07-20T02:45:44.192 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2023-07-20T02:45:44.192 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: expand_channel_meta expand map: {default=false} 2023-07-20T02:45:44.193 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: expand_channel_meta from 'false' to 'false' 2023-07-20T02:45:44.193 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: expand_channel_meta expanded map: {default=false} 2023-07-20T02:45:44.193 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: expand_channel_meta expand map: {default=info} 2023-07-20T02:45:44.193 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: expand_channel_meta from 'info' to 'info' 2023-07-20T02:45:44.193 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: expand_channel_meta expanded map: {default=info} 2023-07-20T02:45:44.193 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: expand_channel_meta expand map: {default=daemon} 2023-07-20T02:45:44.193 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-20T02:45:44.194 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: expand_channel_meta expanded map: {default=daemon} 2023-07-20T02:45:44.194 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: expand_channel_meta expand map: {default=debug} 2023-07-20T02:45:44.194 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: expand_channel_meta from 'debug' to 'debug' 2023-07-20T02:45:44.194 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 ceph-mon[128925]: expand_channel_meta expanded map: {default=debug} 2023-07-20T02:45:44.194 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:43 smithi158 systemd[1]: Started Ceph mon.b for 1b04643c-26a7-11ee-9b34-001a4aab830c. 2023-07-20T02:45:49.209 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:48 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:45:49.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:48 smithi188 ceph-mon[129553]: mon.c calling monitor election 2023-07-20T02:45:49.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:48 smithi188 ceph-mon[129553]: mon.a calling monitor election 2023-07-20T02:45:49.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:48 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:45:49.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:48 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:45:49.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:48 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:45:49.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:48 smithi188 ceph-mon[129553]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:49.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:48 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:45:49.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:48 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:45:49.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:48 smithi188 ceph-mon[129553]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:49.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:48 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:45:49.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:48 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:45:49.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:48 smithi188 ceph-mon[129553]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T02:45:49.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:48 smithi188 ceph-mon[129553]: monmap e3: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0],c=[v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0]} removed_ranks: {} 2023-07-20T02:45:49.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:48 smithi188 ceph-mon[129553]: fsmap 2023-07-20T02:45:49.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:48 smithi188 ceph-mon[129553]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T02:45:49.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:48 smithi188 ceph-mon[129553]: mgrmap e13: x(active, since 55s) 2023-07-20T02:45:49.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:48 smithi188 ceph-mon[129553]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-20T02:45:49.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:48 smithi188 ceph-mon[129553]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-20T02:45:49.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:48 smithi188 ceph-mon[129553]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-20T02:45:49.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:48 smithi188 ceph-mon[129553]: mon.b (rank 2) addr [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] is down (out of quorum) 2023-07-20T02:45:49.215 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:48 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:49.215 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:48 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:49.444 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:48 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:45:49.444 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:48 smithi007 ceph-mon[126353]: mon.c calling monitor election 2023-07-20T02:45:49.444 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:48 smithi007 ceph-mon[126353]: mon.a calling monitor election 2023-07-20T02:45:49.444 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:48 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:45:49.444 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:48 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:45:49.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:48 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:45:49.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:48 smithi007 ceph-mon[126353]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:49.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:48 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:45:49.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:48 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:45:49.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:48 smithi007 ceph-mon[126353]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:49.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:48 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:45:49.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:48 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:45:49.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:48 smithi007 ceph-mon[126353]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T02:45:49.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:48 smithi007 ceph-mon[126353]: monmap e3: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0],c=[v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0]} removed_ranks: {} 2023-07-20T02:45:49.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:48 smithi007 ceph-mon[126353]: fsmap 2023-07-20T02:45:49.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:48 smithi007 ceph-mon[126353]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T02:45:49.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:48 smithi007 ceph-mon[126353]: mgrmap e13: x(active, since 55s) 2023-07-20T02:45:49.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:48 smithi007 ceph-mon[126353]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-20T02:45:49.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:48 smithi007 ceph-mon[126353]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-20T02:45:49.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:48 smithi007 ceph-mon[126353]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-20T02:45:49.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:48 smithi007 ceph-mon[126353]: mon.b (rank 2) addr [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] is down (out of quorum) 2023-07-20T02:45:49.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:48 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:49.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:48 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:49.453 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.445+0000 7f3313e56700 1 -- 172.21.15.188:0/235941526 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32f80035c0 msgr2=0x7f32f8005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:49.454 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.445+0000 7f3313e56700 1 --2- 172.21.15.188:0/235941526 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32f80035c0 0x7f32f8005a50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f330c0ff150 tx=0x7f330800fe60 comp rx=0 tx=0).stop 2023-07-20T02:45:49.454 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.445+0000 7f3313e56700 1 -- 172.21.15.188:0/235941526 shutdown_connections 2023-07-20T02:45:49.454 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.445+0000 7f3313e56700 1 --2- 172.21.15.188:0/235941526 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32f80035c0 0x7f32f8005a50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:49.454 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.445+0000 7f3313e56700 1 -- 172.21.15.188:0/235941526 >> 172.21.15.188:0/235941526 conn(0x7f330c0fad30 msgr2=0x7f330c0fd150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:45:49.454 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.445+0000 7f3313e56700 1 -- 172.21.15.188:0/235941526 shutdown_connections 2023-07-20T02:45:49.454 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.445+0000 7f3313e56700 1 -- 172.21.15.188:0/235941526 wait complete. 2023-07-20T02:45:49.456 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.446+0000 7f3313e56700 1 Processor -- start 2023-07-20T02:45:49.457 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.446+0000 7f3313e56700 1 -- start start 2023-07-20T02:45:49.457 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.446+0000 7f3313e56700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32f80035c0 0x7f330c1945a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:49.457 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.446+0000 7f3313e56700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f330c194ae0 0x7f330c194f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:49.457 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.446+0000 7f3313e56700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f330c1a1c10 0x7f330c1a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:49.457 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.446+0000 7f3313e56700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330c1a4540 con 0x7f32f80035c0 2023-07-20T02:45:49.457 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.446+0000 7f3313e56700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330c1a46b0 con 0x7f330c1a1c10 2023-07-20T02:45:49.457 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.446+0000 7f3313e56700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330c1a4870 con 0x7f330c194ae0 2023-07-20T02:45:49.457 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.446+0000 7f33113f1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f330c194ae0 0x7f330c194f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:49.458 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.446+0000 7f33113f1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f330c194ae0 0x7f330c194f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:34810/0 (socket says 172.21.15.188:34810) 2023-07-20T02:45:49.458 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.446+0000 7f33113f1700 1 -- 172.21.15.188:0/4282922491 learned_addr learned my addr 172.21.15.188:0/4282922491 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-07-20T02:45:49.458 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.447+0000 7f3311bf2700 1 --2- 172.21.15.188:0/4282922491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32f80035c0 0x7f330c1945a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:49.458 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.447+0000 7f33113f1700 1 -- 172.21.15.188:0/4282922491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f330c194ae0 msgr2=0x7f330c194f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-07-20T02:45:49.458 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.447+0000 7f33113f1700 1 -- 172.21.15.188:0/4282922491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f330c194ae0 msgr2=0x7f330c194f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-20T02:45:49.458 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.447+0000 7f33113f1700 1 --2- 172.21.15.188:0/4282922491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f330c194ae0 0x7f330c194f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-20T02:45:49.458 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.447+0000 7f33113f1700 1 --2- 172.21.15.188:0/4282922491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f330c194ae0 0x7f330c194f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T02:45:49.459 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.447+0000 7f33123f3700 1 --2- 172.21.15.188:0/4282922491 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f330c1a1c10 0x7f330c1a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:49.459 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.447+0000 7f33123f3700 1 -- 172.21.15.188:0/4282922491 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f330c1a1c10 msgr2=0x7f330c1a4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2023-07-20T02:45:49.459 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.447+0000 7f33123f3700 1 -- 172.21.15.188:0/4282922491 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f330c1a1c10 msgr2=0x7f330c1a4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-20T02:45:49.459 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.447+0000 7f33123f3700 1 --2- 172.21.15.188:0/4282922491 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f330c1a1c10 0x7f330c1a4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-20T02:45:49.459 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.447+0000 7f3311bf2700 1 -- 172.21.15.188:0/4282922491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f330c194ae0 msgr2=0x7f330c194f60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T02:45:49.459 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.447+0000 7f3311bf2700 1 --2- 172.21.15.188:0/4282922491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f330c194ae0 0x7f330c194f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:49.459 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.447+0000 7f33123f3700 1 --2- 172.21.15.188:0/4282922491 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f330c1a1c10 0x7f330c1a4000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T02:45:49.460 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.447+0000 7f3311bf2700 1 -- 172.21.15.188:0/4282922491 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f330c1a1c10 msgr2=0x7f330c1a4000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T02:45:49.460 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.447+0000 7f3311bf2700 1 --2- 172.21.15.188:0/4282922491 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f330c1a1c10 0x7f330c1a4000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:49.460 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.447+0000 7f3311bf2700 1 -- 172.21.15.188:0/4282922491 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f330800f8b0 con 0x7f32f80035c0 2023-07-20T02:45:49.460 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.447+0000 7f3311bf2700 1 --2- 172.21.15.188:0/4282922491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32f80035c0 0x7f330c1945a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f330c0ff150 tx=0x7f3308012e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:45:49.460 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.447+0000 7f3302ffd700 1 -- 172.21.15.188:0/4282922491 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33080040b0 con 0x7f32f80035c0 2023-07-20T02:45:49.460 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.447+0000 7f3302ffd700 1 -- 172.21.15.188:0/4282922491 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3308004210 con 0x7f32f80035c0 2023-07-20T02:45:49.460 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.447+0000 7f3313e56700 1 -- 172.21.15.188:0/4282922491 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f330c1a4af0 con 0x7f32f80035c0 2023-07-20T02:45:49.460 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.447+0000 7f3302ffd700 1 -- 172.21.15.188:0/4282922491 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33080277f0 con 0x7f32f80035c0 2023-07-20T02:45:49.461 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.447+0000 7f3313e56700 1 -- 172.21.15.188:0/4282922491 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f330c1a4f60 con 0x7f32f80035c0 2023-07-20T02:45:49.461 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.448+0000 7f3302ffd700 1 -- 172.21.15.188:0/4282922491 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f3308024070 con 0x7f32f80035c0 2023-07-20T02:45:49.461 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.448+0000 7f3313e56700 1 -- 172.21.15.188:0/4282922491 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f330c061da0 con 0x7f32f80035c0 2023-07-20T02:45:49.461 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.448+0000 7f3302ffd700 1 --2- 172.21.15.188:0/4282922491 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f32f0038c80 0x7f32f003b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:49.461 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.449+0000 7f3302ffd700 1 -- 172.21.15.188:0/4282922491 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f330804e200 con 0x7f32f80035c0 2023-07-20T02:45:49.462 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.449+0000 7f33113f1700 1 --2- 172.21.15.188:0/4282922491 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f32f0038c80 0x7f32f003b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:49.462 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.449+0000 7f33113f1700 1 --2- 172.21.15.188:0/4282922491 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f32f0038c80 0x7f32f003b140 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f32fc009180 tx=0x7f32fc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:45:49.462 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.454+0000 7f3302ffd700 1 -- 172.21.15.188:0/4282922491 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f330801d440 con 0x7f32f80035c0 2023-07-20T02:45:49.730 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.722+0000 7f3313e56700 1 -- 172.21.15.188:0/4282922491 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f330c1a52b0 con 0x7f32f80035c0 2023-07-20T02:45:49.731 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.723+0000 7f3302ffd700 1 -- 172.21.15.188:0/4282922491 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1270 (secure 0 0 0) 0x7f330802b070 con 0x7f32f80035c0 2023-07-20T02:45:49.731 INFO:teuthology.orchestra.run.smithi188.stdout: 2023-07-20T02:45:49.731 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":3,"fsid":"1b04643c-26a7-11ee-9b34-001a4aab830c","modified":"2023-07-20T02:45:43.883058Z","created":"2023-07-20T02:43:46.393290Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3300","nonce":0},{"type":"v1","addr":"172.21.15.7:6789","nonce":0}]},"addr":"172.21.15.7:6789/0","public_addr":"172.21.15.7:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:3300","nonce":0},{"type":"v1","addr":"172.21.15.188:6789","nonce":0}]},"addr":"172.21.15.188:6789/0","public_addr":"172.21.15.188:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-07-20T02:45:49.733 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.725+0000 7f3313e56700 1 -- 172.21.15.188:0/4282922491 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f32f0038c80 msgr2=0x7f32f003b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:49.734 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.725+0000 7f3313e56700 1 --2- 172.21.15.188:0/4282922491 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f32f0038c80 0x7f32f003b140 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f32fc009180 tx=0x7f32fc008040 comp rx=0 tx=0).stop 2023-07-20T02:45:49.734 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.725+0000 7f3313e56700 1 -- 172.21.15.188:0/4282922491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32f80035c0 msgr2=0x7f330c1945a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:49.734 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.725+0000 7f3313e56700 1 --2- 172.21.15.188:0/4282922491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32f80035c0 0x7f330c1945a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f330c0ff150 tx=0x7f3308012e10 comp rx=0 tx=0).stop 2023-07-20T02:45:49.734 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.725+0000 7f3313e56700 1 -- 172.21.15.188:0/4282922491 shutdown_connections 2023-07-20T02:45:49.734 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.725+0000 7f3313e56700 1 --2- 172.21.15.188:0/4282922491 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f330c1a1c10 0x7f330c1a4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:49.735 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.725+0000 7f3313e56700 1 --2- 172.21.15.188:0/4282922491 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f32f0038c80 0x7f32f003b140 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:49.735 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.725+0000 7f3313e56700 1 --2- 172.21.15.188:0/4282922491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32f80035c0 0x7f330c1945a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:49.735 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.725+0000 7f3313e56700 1 --2- 172.21.15.188:0/4282922491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f330c194ae0 0x7f330c194f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:49.735 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.725+0000 7f3313e56700 1 -- 172.21.15.188:0/4282922491 >> 172.21.15.188:0/4282922491 conn(0x7f330c0fad30 msgr2=0x7f330c0fc2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:45:49.735 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.725+0000 7f3313e56700 1 -- 172.21.15.188:0/4282922491 shutdown_connections 2023-07-20T02:45:49.735 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:49.725+0000 7f3313e56700 1 -- 172.21.15.188:0/4282922491 wait complete. 2023-07-20T02:45:49.736 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 3 2023-07-20T02:45:50.102 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:49 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:50.103 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:49 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:50.103 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:49 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:45:50.103 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:49 smithi188 ceph-mon[129553]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:50.104 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:49 smithi188 ceph-mon[129553]: from='client.? 172.21.15.188:0/4282922491' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T02:45:50.104 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:49 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:45:50.105 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-07-20T02:45:50.106 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph config generate-minimal-conf 2023-07-20T02:45:50.444 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:49 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:50.444 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:49 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:50.444 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:49 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:45:50.444 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:49 smithi007 ceph-mon[126353]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:50.444 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:49 smithi007 ceph-mon[126353]: from='client.? 172.21.15.188:0/4282922491' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T02:45:50.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:49 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:45:50.850 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:45:51.386 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:50 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:45:51.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:50 smithi158 ceph-mon[128925]: mon.c calling monitor election 2023-07-20T02:45:51.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:50 smithi158 ceph-mon[128925]: mon.a calling monitor election 2023-07-20T02:45:51.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:50 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:45:51.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:50 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:45:51.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:50 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:45:51.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:50 smithi158 ceph-mon[128925]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:51.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:50 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:45:51.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:50 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:45:51.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:50 smithi158 ceph-mon[128925]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:51.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:50 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:45:51.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:50 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:45:51.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:50 smithi158 ceph-mon[128925]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T02:45:51.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:50 smithi158 ceph-mon[128925]: monmap e3: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0],c=[v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0]} removed_ranks: {} 2023-07-20T02:45:51.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:50 smithi158 ceph-mon[128925]: fsmap 2023-07-20T02:45:51.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:50 smithi158 ceph-mon[128925]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T02:45:51.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:50 smithi158 ceph-mon[128925]: mgrmap e13: x(active, since 55s) 2023-07-20T02:45:51.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:50 smithi158 ceph-mon[128925]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-20T02:45:51.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:50 smithi158 ceph-mon[128925]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-20T02:45:51.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:50 smithi158 ceph-mon[128925]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-20T02:45:51.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:50 smithi158 ceph-mon[128925]: mon.b (rank 2) addr [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] is down (out of quorum) 2023-07-20T02:45:51.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:50 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:51.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:50 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:51.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:50 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:51.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:50 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:51.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:50 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:45:51.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:50 smithi158 ceph-mon[128925]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:51.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:50 smithi158 ceph-mon[128925]: from='client.? 172.21.15.188:0/4282922491' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T02:45:51.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:50 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:45:52.124 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:51 smithi007 ceph-mon[126353]: mon.b calling monitor election 2023-07-20T02:45:52.125 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:51 smithi007 ceph-mon[126353]: Updating smithi007:/etc/ceph/ceph.conf 2023-07-20T02:45:52.125 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:51 smithi007 ceph-mon[126353]: Updating smithi158:/etc/ceph/ceph.conf 2023-07-20T02:45:52.125 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:51 smithi007 ceph-mon[126353]: Updating smithi188:/etc/ceph/ceph.conf 2023-07-20T02:45:52.125 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:51 smithi007 ceph-mon[126353]: mon.b calling monitor election 2023-07-20T02:45:52.125 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:51 smithi007 ceph-mon[126353]: mon.c calling monitor election 2023-07-20T02:45:52.125 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:51 smithi007 ceph-mon[126353]: mon.a calling monitor election 2023-07-20T02:45:52.125 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:51 smithi007 ceph-mon[126353]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-20T02:45:52.126 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:51 smithi007 ceph-mon[126353]: monmap e3: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0],c=[v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0]} removed_ranks: {} 2023-07-20T02:45:52.126 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:51 smithi007 ceph-mon[126353]: fsmap 2023-07-20T02:45:52.126 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:51 smithi007 ceph-mon[126353]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T02:45:52.126 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:51 smithi007 ceph-mon[126353]: mgrmap e13: x(active, since 57s) 2023-07-20T02:45:52.126 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:51 smithi007 ceph-mon[126353]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-20T02:45:52.126 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:51 smithi007 ceph-mon[126353]: Cluster is now healthy 2023-07-20T02:45:52.126 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:51 smithi007 ceph-mon[126353]: overall HEALTH_OK 2023-07-20T02:45:52.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:51 smithi007 ceph-mon[126353]: Updating smithi158:/var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T02:45:52.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:51 smithi007 ceph-mon[126353]: Updating smithi188:/var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T02:45:52.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:51 smithi007 ceph-mon[126353]: Updating smithi007:/var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T02:45:52.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:51 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:52.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:51 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:52.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:51 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:52.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:51 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:52.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:51 smithi007 ceph-mon[126353]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:52.136 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:51 smithi158 ceph-mon[128925]: mon.b calling monitor election 2023-07-20T02:45:52.136 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:51 smithi158 ceph-mon[128925]: Updating smithi007:/etc/ceph/ceph.conf 2023-07-20T02:45:52.136 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:51 smithi158 ceph-mon[128925]: Updating smithi158:/etc/ceph/ceph.conf 2023-07-20T02:45:52.136 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:51 smithi158 ceph-mon[128925]: Updating smithi188:/etc/ceph/ceph.conf 2023-07-20T02:45:52.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:51 smithi158 ceph-mon[128925]: mon.b calling monitor election 2023-07-20T02:45:52.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:51 smithi158 ceph-mon[128925]: mon.c calling monitor election 2023-07-20T02:45:52.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:51 smithi158 ceph-mon[128925]: mon.a calling monitor election 2023-07-20T02:45:52.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:51 smithi158 ceph-mon[128925]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-20T02:45:52.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:51 smithi158 ceph-mon[128925]: monmap e3: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0],c=[v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0]} removed_ranks: {} 2023-07-20T02:45:52.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:51 smithi158 ceph-mon[128925]: fsmap 2023-07-20T02:45:52.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:51 smithi158 ceph-mon[128925]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T02:45:52.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:51 smithi158 ceph-mon[128925]: mgrmap e13: x(active, since 57s) 2023-07-20T02:45:52.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:51 smithi158 ceph-mon[128925]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-20T02:45:52.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:51 smithi158 ceph-mon[128925]: Cluster is now healthy 2023-07-20T02:45:52.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:51 smithi158 ceph-mon[128925]: overall HEALTH_OK 2023-07-20T02:45:52.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:51 smithi158 ceph-mon[128925]: Updating smithi158:/var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T02:45:52.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:51 smithi158 ceph-mon[128925]: Updating smithi188:/var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T02:45:52.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:51 smithi158 ceph-mon[128925]: Updating smithi007:/var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T02:45:52.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:51 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:52.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:51 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:52.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:51 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:52.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:51 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:52.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:51 smithi158 ceph-mon[128925]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:52.209 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:51 smithi188 ceph-mon[129553]: mon.b calling monitor election 2023-07-20T02:45:52.209 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:51 smithi188 ceph-mon[129553]: Updating smithi007:/etc/ceph/ceph.conf 2023-07-20T02:45:52.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:51 smithi188 ceph-mon[129553]: Updating smithi158:/etc/ceph/ceph.conf 2023-07-20T02:45:52.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:51 smithi188 ceph-mon[129553]: Updating smithi188:/etc/ceph/ceph.conf 2023-07-20T02:45:52.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:51 smithi188 ceph-mon[129553]: mon.b calling monitor election 2023-07-20T02:45:52.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:51 smithi188 ceph-mon[129553]: mon.c calling monitor election 2023-07-20T02:45:52.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:51 smithi188 ceph-mon[129553]: mon.a calling monitor election 2023-07-20T02:45:52.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:51 smithi188 ceph-mon[129553]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-20T02:45:52.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:51 smithi188 ceph-mon[129553]: monmap e3: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0],c=[v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0]} removed_ranks: {} 2023-07-20T02:45:52.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:51 smithi188 ceph-mon[129553]: fsmap 2023-07-20T02:45:52.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:51 smithi188 ceph-mon[129553]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T02:45:52.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:51 smithi188 ceph-mon[129553]: mgrmap e13: x(active, since 57s) 2023-07-20T02:45:52.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:51 smithi188 ceph-mon[129553]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-20T02:45:52.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:51 smithi188 ceph-mon[129553]: Cluster is now healthy 2023-07-20T02:45:52.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:51 smithi188 ceph-mon[129553]: overall HEALTH_OK 2023-07-20T02:45:52.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:51 smithi188 ceph-mon[129553]: Updating smithi158:/var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T02:45:52.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:51 smithi188 ceph-mon[129553]: Updating smithi188:/var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T02:45:52.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:51 smithi188 ceph-mon[129553]: Updating smithi007:/var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T02:45:52.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:51 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:52.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:51 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:52.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:51 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:52.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:51 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:52.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:51 smithi188 ceph-mon[129553]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:52.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.379+0000 7fbc242ae700 1 -- 172.21.15.7:0/3619531174 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc1c100460 msgr2=0x7fbc1c100840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:52.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.379+0000 7fbc242ae700 1 --2- 172.21.15.7:0/3619531174 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc1c100460 0x7fbc1c100840 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbc18009910 tx=0x7fbc18009c20 comp rx=0 tx=0).stop 2023-07-20T02:45:52.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.379+0000 7fbc242ae700 1 -- 172.21.15.7:0/3619531174 shutdown_connections 2023-07-20T02:45:52.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.379+0000 7fbc242ae700 1 --2- 172.21.15.7:0/3619531174 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc1c100460 0x7fbc1c100840 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:52.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.379+0000 7fbc242ae700 1 -- 172.21.15.7:0/3619531174 >> 172.21.15.7:0/3619531174 conn(0x7fbc1c0fbd30 msgr2=0x7fbc1c0fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:45:52.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.379+0000 7fbc242ae700 1 -- 172.21.15.7:0/3619531174 shutdown_connections 2023-07-20T02:45:52.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.379+0000 7fbc242ae700 1 -- 172.21.15.7:0/3619531174 wait complete. 2023-07-20T02:45:52.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.380+0000 7fbc242ae700 1 Processor -- start 2023-07-20T02:45:52.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.381+0000 7fbc242ae700 1 -- start start 2023-07-20T02:45:52.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.381+0000 7fbc242ae700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc1c100460 0x7fbc1c19b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:52.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.381+0000 7fbc242ae700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbc1c19bb50 0x7fbc1c195690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:52.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.381+0000 7fbc242ae700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc1c195bd0 0x7fbc1c196080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:52.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.381+0000 7fbc242ae700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc1c19c1b0 con 0x7fbc1c195bd0 2023-07-20T02:45:52.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.381+0000 7fbc242ae700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc1c1a2430 con 0x7fbc1c19bb50 2023-07-20T02:45:52.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.381+0000 7fbc242ae700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc1c1a25c0 con 0x7fbc1c100460 2023-07-20T02:45:52.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.382+0000 7fbc21849700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbc1c19bb50 0x7fbc1c195690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:52.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.382+0000 7fbc2284b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc1c195bd0 0x7fbc1c196080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:52.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.382+0000 7fbc2204a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc1c100460 0x7fbc1c19b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:52.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.382+0000 7fbc21849700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbc1c19bb50 0x7fbc1c195690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.7:40332/0 (socket says 172.21.15.7:40332) 2023-07-20T02:45:52.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.382+0000 7fbc2284b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc1c195bd0 0x7fbc1c196080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56294/0 (socket says 172.21.15.7:56294) 2023-07-20T02:45:52.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.382+0000 7fbc21849700 1 -- 172.21.15.7:0/361070668 learned_addr learned my addr 172.21.15.7:0/361070668 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:45:52.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.382+0000 7fbc2204a700 1 -- 172.21.15.7:0/361070668 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbc1c19bb50 msgr2=0x7fbc1c195690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:52.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.382+0000 7fbc2204a700 1 --2- 172.21.15.7:0/361070668 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbc1c19bb50 0x7fbc1c195690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:52.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.382+0000 7fbc2204a700 1 -- 172.21.15.7:0/361070668 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc1c195bd0 msgr2=0x7fbc1c196080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:52.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.382+0000 7fbc2204a700 1 --2- 172.21.15.7:0/361070668 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc1c195bd0 0x7fbc1c196080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:52.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.382+0000 7fbc2204a700 1 -- 172.21.15.7:0/361070668 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc180095f0 con 0x7fbc1c100460 2023-07-20T02:45:52.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.382+0000 7fbc2284b700 1 --2- 172.21.15.7:0/361070668 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc1c195bd0 0x7fbc1c196080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T02:45:52.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.383+0000 7fbc2204a700 1 --2- 172.21.15.7:0/361070668 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc1c100460 0x7fbc1c19b610 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbc18005950 tx=0x7fbc18004cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:45:52.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.383+0000 7fbc137fe700 1 -- 172.21.15.7:0/361070668 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc1801d070 con 0x7fbc1c100460 2023-07-20T02:45:52.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.383+0000 7fbc137fe700 1 -- 172.21.15.7:0/361070668 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbc18005000 con 0x7fbc1c100460 2023-07-20T02:45:52.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.383+0000 7fbc242ae700 1 -- 172.21.15.7:0/361070668 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc1c1a2840 con 0x7fbc1c100460 2023-07-20T02:45:52.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.383+0000 7fbc137fe700 1 -- 172.21.15.7:0/361070668 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc18022d70 con 0x7fbc1c100460 2023-07-20T02:45:52.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.383+0000 7fbc242ae700 1 -- 172.21.15.7:0/361070668 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc1c1a2d30 con 0x7fbc1c100460 2023-07-20T02:45:52.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.384+0000 7fbc137fe700 1 -- 172.21.15.7:0/361070668 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7fbc1800f460 con 0x7fbc1c100460 2023-07-20T02:45:52.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.384+0000 7fbc242ae700 1 -- 172.21.15.7:0/361070668 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc1c061da0 con 0x7fbc1c100460 2023-07-20T02:45:52.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.385+0000 7fbc137fe700 1 --2- 172.21.15.7:0/361070668 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fbc08038c80 0x7fbc0803b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:52.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.385+0000 7fbc137fe700 1 -- 172.21.15.7:0/361070668 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fbc1804c780 con 0x7fbc1c100460 2023-07-20T02:45:52.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.385+0000 7fbc21849700 1 --2- 172.21.15.7:0/361070668 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fbc08038c80 0x7fbc0803b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:52.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.385+0000 7fbc21849700 1 --2- 172.21.15.7:0/361070668 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fbc08038c80 0x7fbc0803b140 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c009b80 tx=0x7fbc0c0092c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:45:52.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.390+0000 7fbc137fe700 1 -- 172.21.15.7:0/361070668 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fbc1800e3e0 con 0x7fbc1c100460 2023-07-20T02:45:52.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.599+0000 7fbc242ae700 1 -- 172.21.15.7:0/361070668 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fbc1c196ba0 con 0x7fbc1c100460 2023-07-20T02:45:52.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.600+0000 7fbc137fe700 1 -- 172.21.15.7:0/361070668 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+273 (secure 0 0 0) 0x7fbc18027020 con 0x7fbc1c100460 2023-07-20T02:45:52.609 INFO:teuthology.orchestra.run.smithi007.stdout:# minimal ceph.conf for 1b04643c-26a7-11ee-9b34-001a4aab830c 2023-07-20T02:45:52.609 INFO:teuthology.orchestra.run.smithi007.stdout:[global] 2023-07-20T02:45:52.610 INFO:teuthology.orchestra.run.smithi007.stdout: fsid = 1b04643c-26a7-11ee-9b34-001a4aab830c 2023-07-20T02:45:52.610 INFO:teuthology.orchestra.run.smithi007.stdout: mon_host = [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] 2023-07-20T02:45:52.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.602+0000 7fbc242ae700 1 -- 172.21.15.7:0/361070668 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fbc08038c80 msgr2=0x7fbc0803b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:52.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.602+0000 7fbc242ae700 1 --2- 172.21.15.7:0/361070668 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fbc08038c80 0x7fbc0803b140 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c009b80 tx=0x7fbc0c0092c0 comp rx=0 tx=0).stop 2023-07-20T02:45:52.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.602+0000 7fbc242ae700 1 -- 172.21.15.7:0/361070668 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc1c100460 msgr2=0x7fbc1c19b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:52.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.602+0000 7fbc242ae700 1 --2- 172.21.15.7:0/361070668 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc1c100460 0x7fbc1c19b610 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbc18005950 tx=0x7fbc18004cb0 comp rx=0 tx=0).stop 2023-07-20T02:45:52.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.602+0000 7fbc242ae700 1 -- 172.21.15.7:0/361070668 shutdown_connections 2023-07-20T02:45:52.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.602+0000 7fbc242ae700 1 --2- 172.21.15.7:0/361070668 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fbc08038c80 0x7fbc0803b140 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:52.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.602+0000 7fbc242ae700 1 --2- 172.21.15.7:0/361070668 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc1c195bd0 0x7fbc1c196080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:52.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.602+0000 7fbc242ae700 1 --2- 172.21.15.7:0/361070668 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc1c100460 0x7fbc1c19b610 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:52.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.602+0000 7fbc242ae700 1 --2- 172.21.15.7:0/361070668 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbc1c19bb50 0x7fbc1c195690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:52.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.602+0000 7fbc242ae700 1 -- 172.21.15.7:0/361070668 >> 172.21.15.7:0/361070668 conn(0x7fbc1c0fbd30 msgr2=0x7fbc1c105fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:45:52.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.602+0000 7fbc242ae700 1 -- 172.21.15.7:0/361070668 shutdown_connections 2023-07-20T02:45:52.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:45:52.602+0000 7fbc242ae700 1 -- 172.21.15.7:0/361070668 wait complete. 2023-07-20T02:45:52.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:52 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:45:52.893 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:52 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:52.893 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:52 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:52.893 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:52 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:52.893 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:52 smithi007 ceph-mon[126353]: Reconfiguring mon.a (unknown last config time)... 2023-07-20T02:45:52.893 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:52 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T02:45:52.894 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:52 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T02:45:52.894 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:52 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:45:52.895 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:52 smithi007 ceph-mon[126353]: Reconfiguring daemon mon.a on smithi007 2023-07-20T02:45:52.895 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:52 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/361070668' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:45:53.072 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-07-20T02:45:53.073 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-07-20T02:45:53.073 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.conf 2023-07-20T02:45:53.136 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:52 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:45:53.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:52 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:53.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:52 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:53.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:52 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:53.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:52 smithi158 ceph-mon[128925]: Reconfiguring mon.a (unknown last config time)... 2023-07-20T02:45:53.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:52 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T02:45:53.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:52 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T02:45:53.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:52 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:45:53.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:52 smithi158 ceph-mon[128925]: Reconfiguring daemon mon.a on smithi007 2023-07-20T02:45:53.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:52 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/361070668' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:45:53.156 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:45:52 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:45:52.882+0000 7f610e4dd700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-07-20T02:45:53.164 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-07-20T02:45:53.164 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T02:45:53.209 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:52 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:45:53.209 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:52 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:53.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:52 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:53.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:52 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:53.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:52 smithi188 ceph-mon[129553]: Reconfiguring mon.a (unknown last config time)... 2023-07-20T02:45:53.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:52 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T02:45:53.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:52 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T02:45:53.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:52 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:45:53.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:52 smithi188 ceph-mon[129553]: Reconfiguring daemon mon.a on smithi007 2023-07-20T02:45:53.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:52 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/361070668' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:45:53.275 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-07-20T02:45:53.276 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/etc/ceph/ceph.conf 2023-07-20T02:45:53.312 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-07-20T02:45:53.313 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T02:45:53.384 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2023-07-20T02:45:53.384 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/ceph/ceph.conf 2023-07-20T02:45:53.421 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2023-07-20T02:45:53.421 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T02:45:53.491 INFO:tasks.cephadm:Adding mgr.x on smithi007 2023-07-20T02:45:53.491 INFO:tasks.cephadm:Adding mgr.y on smithi158 2023-07-20T02:45:53.491 INFO:tasks.cephadm:Adding mgr.z on smithi188 2023-07-20T02:45:53.492 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph orch apply mgr '3;smithi007=x;smithi158=y;smithi188=z' 2023-07-20T02:45:54.082 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T02:45:54.701 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:54 smithi158 ceph-mon[128925]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:54.701 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:54 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:54.702 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:54 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:54.702 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:54 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:54.702 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:54 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T02:45:54.703 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:54 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T02:45:54.703 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:54 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:45:54.709 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:54 smithi188 ceph-mon[129553]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:54.709 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:54 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:54.709 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:54 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:54.710 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:54 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:54.710 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:54 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T02:45:54.710 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:54 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T02:45:54.710 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:54 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:45:54.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:54 smithi007 ceph-mon[126353]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:54.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:54 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:54.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:54 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:54.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:54 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:54.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:54 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T02:45:54.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:54 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T02:45:54.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:54 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:45:55.627 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.618+0000 7f99601b8700 1 -- 172.21.15.188:0/1817397077 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99580ff290 msgr2=0x7f99580ff6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:55.628 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.618+0000 7f99601b8700 1 --2- 172.21.15.188:0/1817397077 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99580ff290 0x7f99580ff6b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9950009910 tx=0x7f9950009c20 comp rx=0 tx=0).stop 2023-07-20T02:45:55.628 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.619+0000 7f99601b8700 1 -- 172.21.15.188:0/1817397077 shutdown_connections 2023-07-20T02:45:55.628 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.619+0000 7f99601b8700 1 --2- 172.21.15.188:0/1817397077 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99580ff290 0x7f99580ff6b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:55.628 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.619+0000 7f99601b8700 1 -- 172.21.15.188:0/1817397077 >> 172.21.15.188:0/1817397077 conn(0x7f99580fa810 msgr2=0x7f99580fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:45:55.628 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.619+0000 7f99601b8700 1 -- 172.21.15.188:0/1817397077 shutdown_connections 2023-07-20T02:45:55.628 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.619+0000 7f99601b8700 1 -- 172.21.15.188:0/1817397077 wait complete. 2023-07-20T02:45:55.628 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.619+0000 7f99601b8700 1 Processor -- start 2023-07-20T02:45:55.629 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.619+0000 7f99601b8700 1 -- start start 2023-07-20T02:45:55.630 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.619+0000 7f99601b8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99580ff290 0x7f9958194500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:55.630 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.619+0000 7f99601b8700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9958194a40 0x7f9958199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:55.630 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.619+0000 7f99601b8700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9958194ec0 0x7f995819c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:55.631 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.619+0000 7f99601b8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f995819c7e0 con 0x7f99580ff290 2023-07-20T02:45:55.631 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.619+0000 7f99601b8700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f995819c950 con 0x7f9958194a40 2023-07-20T02:45:55.631 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.619+0000 7f99601b8700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f995819cb10 con 0x7f9958194ec0 2023-07-20T02:45:55.631 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.620+0000 7f995e755700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9958194ec0 0x7f995819c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:55.631 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.620+0000 7f995e755700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9958194ec0 0x7f995819c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:34822/0 (socket says 172.21.15.188:34822) 2023-07-20T02:45:55.631 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.620+0000 7f995e755700 1 -- 172.21.15.188:0/3578929710 learned_addr learned my addr 172.21.15.188:0/3578929710 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-07-20T02:45:55.631 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.620+0000 7f995d753700 1 --2- 172.21.15.188:0/3578929710 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9958194a40 0x7f9958199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:55.632 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.620+0000 7f995df54700 1 --2- 172.21.15.188:0/3578929710 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99580ff290 0x7f9958194500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:55.632 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.620+0000 7f995e755700 1 -- 172.21.15.188:0/3578929710 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9958194a40 msgr2=0x7f9958199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:55.632 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.620+0000 7f995e755700 1 --2- 172.21.15.188:0/3578929710 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9958194a40 0x7f9958199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:55.632 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.620+0000 7f995e755700 1 -- 172.21.15.188:0/3578929710 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99580ff290 msgr2=0x7f9958194500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:55.632 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.620+0000 7f995e755700 1 --2- 172.21.15.188:0/3578929710 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99580ff290 0x7f9958194500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:55.632 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.620+0000 7f995e755700 1 -- 172.21.15.188:0/3578929710 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99500095f0 con 0x7f9958194ec0 2023-07-20T02:45:55.632 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.620+0000 7f995e755700 1 --2- 172.21.15.188:0/3578929710 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9958194ec0 0x7f995819c040 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f994800ba50 tx=0x7f994800bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:45:55.633 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.620+0000 7f9956ffd700 1 -- 172.21.15.188:0/3578929710 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994800c9b0 con 0x7f9958194ec0 2023-07-20T02:45:55.633 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.620+0000 7f9956ffd700 1 -- 172.21.15.188:0/3578929710 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9948014440 con 0x7f9958194ec0 2023-07-20T02:45:55.633 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.620+0000 7f9956ffd700 1 -- 172.21.15.188:0/3578929710 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9948007590 con 0x7f9958194ec0 2023-07-20T02:45:55.633 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.620+0000 7f99601b8700 1 -- 172.21.15.188:0/3578929710 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f995819cdf0 con 0x7f9958194ec0 2023-07-20T02:45:55.633 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.620+0000 7f99601b8700 1 -- 172.21.15.188:0/3578929710 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f995819d2e0 con 0x7f9958194ec0 2023-07-20T02:45:55.633 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.621+0000 7f9956ffd700 1 -- 172.21.15.188:0/3578929710 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f99480145b0 con 0x7f9958194ec0 2023-07-20T02:45:55.633 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.621+0000 7f99601b8700 1 -- 172.21.15.188:0/3578929710 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f995818dd50 con 0x7f9958194ec0 2023-07-20T02:45:55.633 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.621+0000 7f9956ffd700 1 --2- 172.21.15.188:0/3578929710 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f9940038c30 0x7f994003b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:45:55.634 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.621+0000 7f9956ffd700 1 -- 172.21.15.188:0/3578929710 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f994804a970 con 0x7f9958194ec0 2023-07-20T02:45:55.634 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.622+0000 7f995df54700 1 --2- 172.21.15.188:0/3578929710 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f9940038c30 0x7f994003b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:45:55.634 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.622+0000 7f995df54700 1 --2- 172.21.15.188:0/3578929710 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f9940038c30 0x7f994003b0f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9950005130 tx=0x7f995000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:45:55.634 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.626+0000 7f9956ffd700 1 -- 172.21.15.188:0/3578929710 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f994800cb10 con 0x7f9958194ec0 2023-07-20T02:45:55.709 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:55 smithi188 ceph-mon[129553]: Reconfiguring mon.b (monmap changed)... 2023-07-20T02:45:55.710 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:55 smithi188 ceph-mon[129553]: Reconfiguring daemon mon.b on smithi158 2023-07-20T02:45:55.710 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:55 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:55.710 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:55 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:55.710 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:55 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T02:45:55.710 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:55 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T02:45:55.710 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:55 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:45:55.851 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.843+0000 7f99601b8700 1 -- 172.21.15.188:0/3578929710 --> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi007=x;smithi158=y;smithi188=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f9958060bb0 con 0x7f9940038c30 2023-07-20T02:45:55.858 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.850+0000 7f9956ffd700 1 -- 172.21.15.188:0/3578929710 <== mgr.14152 v2:172.21.15.7:6800/1354542137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f9958060bb0 con 0x7f9940038c30 2023-07-20T02:45:55.858 INFO:teuthology.orchestra.run.smithi188.stdout:Scheduled mgr update... 2023-07-20T02:45:55.860 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.852+0000 7f99601b8700 1 -- 172.21.15.188:0/3578929710 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f9940038c30 msgr2=0x7f994003b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:55.860 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.852+0000 7f99601b8700 1 --2- 172.21.15.188:0/3578929710 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f9940038c30 0x7f994003b0f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9950005130 tx=0x7f995000b540 comp rx=0 tx=0).stop 2023-07-20T02:45:55.860 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.852+0000 7f99601b8700 1 -- 172.21.15.188:0/3578929710 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9958194ec0 msgr2=0x7f995819c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:45:55.860 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.852+0000 7f99601b8700 1 --2- 172.21.15.188:0/3578929710 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9958194ec0 0x7f995819c040 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f994800ba50 tx=0x7f994800bd60 comp rx=0 tx=0).stop 2023-07-20T02:45:55.861 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.852+0000 7f99601b8700 1 -- 172.21.15.188:0/3578929710 shutdown_connections 2023-07-20T02:45:55.861 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.852+0000 7f99601b8700 1 --2- 172.21.15.188:0/3578929710 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9958194ec0 0x7f995819c040 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:55.861 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.852+0000 7f99601b8700 1 --2- 172.21.15.188:0/3578929710 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f9940038c30 0x7f994003b0f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:55.861 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.852+0000 7f99601b8700 1 --2- 172.21.15.188:0/3578929710 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99580ff290 0x7f9958194500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:55.861 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.852+0000 7f99601b8700 1 --2- 172.21.15.188:0/3578929710 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9958194a40 0x7f9958199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:45:55.861 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.852+0000 7f99601b8700 1 -- 172.21.15.188:0/3578929710 >> 172.21.15.188:0/3578929710 conn(0x7f99580fa810 msgr2=0x7f99580fb550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:45:55.861 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.852+0000 7f99601b8700 1 -- 172.21.15.188:0/3578929710 shutdown_connections 2023-07-20T02:45:55.862 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:45:55.852+0000 7f99601b8700 1 -- 172.21.15.188:0/3578929710 wait complete. 2023-07-20T02:45:55.886 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:55 smithi158 ceph-mon[128925]: Reconfiguring mon.b (monmap changed)... 2023-07-20T02:45:55.886 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:55 smithi158 ceph-mon[128925]: Reconfiguring daemon mon.b on smithi158 2023-07-20T02:45:55.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:55 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:55.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:55 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:55.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:55 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T02:45:55.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:55 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T02:45:55.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:55 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:45:55.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:55 smithi007 ceph-mon[126353]: Reconfiguring mon.b (monmap changed)... 2023-07-20T02:45:55.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:55 smithi007 ceph-mon[126353]: Reconfiguring daemon mon.b on smithi158 2023-07-20T02:45:55.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:55 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:55.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:55 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:55.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:55 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T02:45:55.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:55 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T02:45:55.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:55 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:45:56.438 DEBUG:teuthology.orchestra.run.smithi158:mgr.y> sudo journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mgr.y.service 2023-07-20T02:45:56.441 DEBUG:teuthology.orchestra.run.smithi188:mgr.z> sudo journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mgr.z.service 2023-07-20T02:45:56.444 INFO:tasks.cephadm:Deploying OSDs... 2023-07-20T02:45:56.444 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-07-20T02:45:56.444 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2023-07-20T02:45:56.473 INFO:journalctl@ceph.mgr.z.smithi188.stdout:-- Logs begin at Thu 2023-07-20 02:09:43 UTC. -- 2023-07-20T02:45:56.475 INFO:journalctl@ceph.mgr.y.smithi158.stdout:-- Logs begin at Thu 2023-07-20 02:10:12 UTC. -- 2023-07-20T02:45:56.493 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-20T02:45:56.494 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_1 2023-07-20T02:45:56.581 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-20T02:45:56.581 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T02:45:56.582 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 446 Links: 1 2023-07-20T02:45:56.582 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T02:45:56.582 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T02:45:56.582 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2023-07-20 02:45:04.863042450 +0000 2023-07-20T02:45:56.583 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2023-07-20 02:37:43.095223420 +0000 2023-07-20T02:45:56.583 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2023-07-20 02:37:43.095223420 +0000 2023-07-20T02:45:56.583 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2023-07-20 02:37:43.072223778 +0000 2023-07-20T02:45:56.584 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-20T02:45:56.652 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2023-07-20T02:45:56.653 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2023-07-20T02:45:56.653 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.0001917 s, 2.7 MB/s 2023-07-20T02:45:56.655 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-20T02:45:56.701 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_2 2023-07-20T02:45:56.709 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:56 smithi188 ceph-mon[129553]: Reconfiguring mon.c (monmap changed)... 2023-07-20T02:45:56.710 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:56 smithi188 ceph-mon[129553]: Reconfiguring daemon mon.c on smithi188 2023-07-20T02:45:56.710 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:56 smithi188 ceph-mon[129553]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:56.711 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:56 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:56.786 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-20T02:45:56.786 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T02:45:56.786 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 452 Links: 1 2023-07-20T02:45:56.787 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T02:45:56.787 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T02:45:56.787 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2023-07-20 02:45:04.863042450 +0000 2023-07-20T02:45:56.787 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2023-07-20 02:37:43.951210079 +0000 2023-07-20T02:45:56.787 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2023-07-20 02:37:43.951210079 +0000 2023-07-20T02:45:56.787 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2023-07-20 02:37:43.929210422 +0000 2023-07-20T02:45:56.788 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-20T02:45:56.886 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:56 smithi158 ceph-mon[128925]: Reconfiguring mon.c (monmap changed)... 2023-07-20T02:45:56.886 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:56 smithi158 ceph-mon[128925]: Reconfiguring daemon mon.c on smithi188 2023-07-20T02:45:56.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:56 smithi158 ceph-mon[128925]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:56.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:56 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:56.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:56 smithi007 ceph-mon[126353]: Reconfiguring mon.c (monmap changed)... 2023-07-20T02:45:56.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:56 smithi007 ceph-mon[126353]: Reconfiguring daemon mon.c on smithi188 2023-07-20T02:45:56.890 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:56 smithi007 ceph-mon[126353]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:56.890 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:56 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:56.896 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2023-07-20T02:45:56.896 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2023-07-20T02:45:56.896 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000189019 s, 2.7 MB/s 2023-07-20T02:45:56.899 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-20T02:45:56.946 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_3 2023-07-20T02:45:56.991 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-20T02:45:56.991 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T02:45:56.991 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 458 Links: 1 2023-07-20T02:45:56.991 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T02:45:56.991 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T02:45:56.992 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2023-07-20 02:45:04.863042450 +0000 2023-07-20T02:45:56.992 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2023-07-20 02:37:44.805196770 +0000 2023-07-20T02:45:56.992 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2023-07-20 02:37:44.805196770 +0000 2023-07-20T02:45:56.992 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2023-07-20 02:37:44.781197144 +0000 2023-07-20T02:45:56.992 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-20T02:45:57.098 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2023-07-20T02:45:57.099 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2023-07-20T02:45:57.099 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000208829 s, 2.5 MB/s 2023-07-20T02:45:57.101 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-20T02:45:57.146 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_4 2023-07-20T02:45:57.232 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-20T02:45:57.232 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T02:45:57.232 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 464 Links: 1 2023-07-20T02:45:57.232 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T02:45:57.233 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T02:45:57.233 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2023-07-20 02:45:04.863042450 +0000 2023-07-20T02:45:57.233 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2023-07-20 02:37:45.669183303 +0000 2023-07-20T02:45:57.233 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2023-07-20 02:37:45.669183303 +0000 2023-07-20T02:45:57.233 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2023-07-20 02:37:45.647183646 +0000 2023-07-20T02:45:57.239 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-20T02:45:57.339 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2023-07-20T02:45:57.339 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2023-07-20T02:45:57.340 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000190406 s, 2.7 MB/s 2023-07-20T02:45:57.342 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-20T02:45:57.387 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-07-20T02:45:57.387 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/scratch_devs of=/dev/stdout 2023-07-20T02:45:57.409 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-20T02:45:57.409 DEBUG:teuthology.orchestra.run.smithi158:> stat /dev/vg_nvme/lv_1 2023-07-20T02:45:57.468 INFO:teuthology.orchestra.run.smithi158.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-20T02:45:57.469 INFO:teuthology.orchestra.run.smithi158.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T02:45:57.469 INFO:teuthology.orchestra.run.smithi158.stdout:Device: 6h/6d Inode: 380058 Links: 1 2023-07-20T02:45:57.469 INFO:teuthology.orchestra.run.smithi158.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T02:45:57.469 INFO:teuthology.orchestra.run.smithi158.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T02:45:57.469 INFO:teuthology.orchestra.run.smithi158.stdout:Access: 2023-07-20 02:45:43.191001913 +0000 2023-07-20T02:45:57.469 INFO:teuthology.orchestra.run.smithi158.stdout:Modify: 2023-07-20 02:32:19.832226389 +0000 2023-07-20T02:45:57.470 INFO:teuthology.orchestra.run.smithi158.stdout:Change: 2023-07-20 02:32:19.832226389 +0000 2023-07-20T02:45:57.470 INFO:teuthology.orchestra.run.smithi158.stdout: Birth: - 2023-07-20T02:45:57.470 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-20T02:45:57.537 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records in 2023-07-20T02:45:57.537 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records out 2023-07-20T02:45:57.538 INFO:teuthology.orchestra.run.smithi158.stderr:512 bytes copied, 0.00012893 s, 4.0 MB/s 2023-07-20T02:45:57.539 DEBUG:teuthology.orchestra.run.smithi158:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-20T02:45:57.599 DEBUG:teuthology.orchestra.run.smithi158:> stat /dev/vg_nvme/lv_2 2023-07-20T02:45:57.658 INFO:teuthology.orchestra.run.smithi158.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-20T02:45:57.659 INFO:teuthology.orchestra.run.smithi158.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T02:45:57.659 INFO:teuthology.orchestra.run.smithi158.stdout:Device: 6h/6d Inode: 380190 Links: 1 2023-07-20T02:45:57.659 INFO:teuthology.orchestra.run.smithi158.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T02:45:57.659 INFO:teuthology.orchestra.run.smithi158.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T02:45:57.659 INFO:teuthology.orchestra.run.smithi158.stdout:Access: 2023-07-20 02:45:43.192001901 +0000 2023-07-20T02:45:57.659 INFO:teuthology.orchestra.run.smithi158.stdout:Modify: 2023-07-20 02:32:20.403219866 +0000 2023-07-20T02:45:57.659 INFO:teuthology.orchestra.run.smithi158.stdout:Change: 2023-07-20 02:32:20.403219866 +0000 2023-07-20T02:45:57.660 INFO:teuthology.orchestra.run.smithi158.stdout: Birth: - 2023-07-20T02:45:57.660 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-20T02:45:57.694 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:57 smithi007 ceph-mon[126353]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi007=x;smithi158=y;smithi188=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:45:57.694 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:57 smithi007 ceph-mon[126353]: Saving service mgr spec with placement smithi007=x;smithi158=y;smithi188=z;count:3 2023-07-20T02:45:57.694 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:57 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:57.694 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:57 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:57.694 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:57 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:45:57.726 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records in 2023-07-20T02:45:57.726 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records out 2023-07-20T02:45:57.726 INFO:teuthology.orchestra.run.smithi158.stderr:512 bytes copied, 0.000108929 s, 4.7 MB/s 2023-07-20T02:45:57.728 DEBUG:teuthology.orchestra.run.smithi158:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-20T02:45:57.786 DEBUG:teuthology.orchestra.run.smithi158:> stat /dev/vg_nvme/lv_3 2023-07-20T02:45:57.844 INFO:teuthology.orchestra.run.smithi158.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-20T02:45:57.844 INFO:teuthology.orchestra.run.smithi158.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T02:45:57.844 INFO:teuthology.orchestra.run.smithi158.stdout:Device: 6h/6d Inode: 380234 Links: 1 2023-07-20T02:45:57.845 INFO:teuthology.orchestra.run.smithi158.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T02:45:57.845 INFO:teuthology.orchestra.run.smithi158.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T02:45:57.845 INFO:teuthology.orchestra.run.smithi158.stdout:Access: 2023-07-20 02:45:43.192001901 +0000 2023-07-20T02:45:57.845 INFO:teuthology.orchestra.run.smithi158.stdout:Modify: 2023-07-20 02:32:20.956213548 +0000 2023-07-20T02:45:57.846 INFO:teuthology.orchestra.run.smithi158.stdout:Change: 2023-07-20 02:32:20.956213548 +0000 2023-07-20T02:45:57.846 INFO:teuthology.orchestra.run.smithi158.stdout: Birth: - 2023-07-20T02:45:57.846 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-20T02:45:57.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:57 smithi158 ceph-mon[128925]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi007=x;smithi158=y;smithi188=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:45:57.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:57 smithi158 ceph-mon[128925]: Saving service mgr spec with placement smithi007=x;smithi158=y;smithi188=z;count:3 2023-07-20T02:45:57.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:57 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:57.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:57 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:57.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:57 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:45:57.910 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records in 2023-07-20T02:45:57.910 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records out 2023-07-20T02:45:57.911 INFO:teuthology.orchestra.run.smithi158.stderr:512 bytes copied, 0.000113304 s, 4.5 MB/s 2023-07-20T02:45:57.912 DEBUG:teuthology.orchestra.run.smithi158:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-20T02:45:57.959 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:57 smithi188 ceph-mon[129553]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi007=x;smithi158=y;smithi188=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:45:57.959 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:57 smithi188 ceph-mon[129553]: Saving service mgr spec with placement smithi007=x;smithi158=y;smithi188=z;count:3 2023-07-20T02:45:57.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:57 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:57.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:57 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:57.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:57 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:45:57.970 DEBUG:teuthology.orchestra.run.smithi158:> stat /dev/vg_nvme/lv_4 2023-07-20T02:45:58.027 INFO:teuthology.orchestra.run.smithi158.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-20T02:45:58.027 INFO:teuthology.orchestra.run.smithi158.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T02:45:58.027 INFO:teuthology.orchestra.run.smithi158.stdout:Device: 6h/6d Inode: 382201 Links: 1 2023-07-20T02:45:58.028 INFO:teuthology.orchestra.run.smithi158.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T02:45:58.028 INFO:teuthology.orchestra.run.smithi158.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T02:45:58.028 INFO:teuthology.orchestra.run.smithi158.stdout:Access: 2023-07-20 02:45:43.192001901 +0000 2023-07-20T02:45:58.028 INFO:teuthology.orchestra.run.smithi158.stdout:Modify: 2023-07-20 02:32:21.472207653 +0000 2023-07-20T02:45:58.028 INFO:teuthology.orchestra.run.smithi158.stdout:Change: 2023-07-20 02:32:21.472207653 +0000 2023-07-20T02:45:58.028 INFO:teuthology.orchestra.run.smithi158.stdout: Birth: - 2023-07-20T02:45:58.029 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-20T02:45:58.093 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records in 2023-07-20T02:45:58.093 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records out 2023-07-20T02:45:58.094 INFO:teuthology.orchestra.run.smithi158.stderr:512 bytes copied, 9.5891e-05 s, 5.3 MB/s 2023-07-20T02:45:58.095 DEBUG:teuthology.orchestra.run.smithi158:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-20T02:45:58.152 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2023-07-20T02:45:58.153 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/scratch_devs of=/dev/stdout 2023-07-20T02:45:58.170 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-20T02:45:58.170 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_1 2023-07-20T02:45:58.227 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-20T02:45:58.228 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T02:45:58.228 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 6h/6d Inode: 383089 Links: 1 2023-07-20T02:45:58.228 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T02:45:58.228 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T02:45:58.228 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2023-07-20 02:45:34.681870190 +0000 2023-07-20T02:45:58.228 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2023-07-20 02:32:22.053240943 +0000 2023-07-20T02:45:58.228 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2023-07-20 02:32:22.053240943 +0000 2023-07-20T02:45:58.229 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: - 2023-07-20T02:45:58.229 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-20T02:45:58.294 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2023-07-20T02:45:58.294 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2023-07-20T02:45:58.294 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000257276 s, 2.0 MB/s 2023-07-20T02:45:58.295 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-20T02:45:58.353 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_2 2023-07-20T02:45:58.411 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-20T02:45:58.412 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T02:45:58.412 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 6h/6d Inode: 377162 Links: 1 2023-07-20T02:45:58.412 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T02:45:58.412 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T02:45:58.412 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2023-07-20 02:45:34.681870190 +0000 2023-07-20T02:45:58.412 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2023-07-20 02:32:22.555231918 +0000 2023-07-20T02:45:58.412 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2023-07-20 02:32:22.555231918 +0000 2023-07-20T02:45:58.413 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: - 2023-07-20T02:45:58.413 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-20T02:45:58.477 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2023-07-20T02:45:58.478 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2023-07-20T02:45:58.478 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000254416 s, 2.0 MB/s 2023-07-20T02:45:58.479 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-20T02:45:58.538 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_3 2023-07-20T02:45:58.595 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-20T02:45:58.596 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T02:45:58.596 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 6h/6d Inode: 377254 Links: 1 2023-07-20T02:45:58.597 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T02:45:58.597 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T02:45:58.597 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2023-07-20 02:45:34.681870190 +0000 2023-07-20T02:45:58.597 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2023-07-20 02:32:23.114221869 +0000 2023-07-20T02:45:58.598 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2023-07-20 02:32:23.114221869 +0000 2023-07-20T02:45:58.598 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: - 2023-07-20T02:45:58.598 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-20T02:45:58.660 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:58 smithi188 ceph-mon[129553]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:58.660 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:58 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:45:58.660 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:58 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:45:58.660 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:58 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:58.660 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:58 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T02:45:58.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:58 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T02:45:58.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:58 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T02:45:58.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:58 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:45:58.667 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2023-07-20T02:45:58.668 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2023-07-20T02:45:58.668 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.00023676 s, 2.2 MB/s 2023-07-20T02:45:58.669 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-20T02:45:58.726 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_4 2023-07-20T02:45:58.782 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-20T02:45:58.782 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T02:45:58.783 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 6h/6d Inode: 377353 Links: 1 2023-07-20T02:45:58.783 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T02:45:58.783 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T02:45:58.783 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2023-07-20 02:45:34.681870190 +0000 2023-07-20T02:45:58.783 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2023-07-20 02:32:23.646212305 +0000 2023-07-20T02:45:58.783 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2023-07-20 02:32:23.646212305 +0000 2023-07-20T02:45:58.783 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: - 2023-07-20T02:45:58.784 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-20T02:45:58.848 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2023-07-20T02:45:58.848 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2023-07-20T02:45:58.848 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000169304 s, 3.0 MB/s 2023-07-20T02:45:58.849 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-20T02:45:58.886 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:58 smithi158 ceph-mon[128925]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:58.886 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:58 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:45:58.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:58 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:45:58.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:58 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:58.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:58 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T02:45:58.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:58 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T02:45:58.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:58 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T02:45:58.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:58 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:45:58.906 INFO:tasks.cephadm:Deploying osd.0 on smithi007 with /dev/vg_nvme/lv_4... 2023-07-20T02:45:58.907 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-20T02:45:58.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:58 smithi007 ceph-mon[126353]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:45:58.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:58 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:45:58.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:58 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:45:58.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:58 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:45:58.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:58 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T02:45:58.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:58 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T02:45:58.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:58 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T02:45:58.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:58 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:45:59.459 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 20 02:45:59 smithi188 systemd[1]: Starting Ceph mgr.z for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:45:59.638 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:45:59.785 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:45:59 smithi188 ceph-mon[129553]: Deploying daemon mgr.z on smithi188 2023-07-20T02:45:59.786 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 20 02:45:59 smithi188 podman[131657]: 2023-07-20T02:45:59.786 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 20 02:45:59 smithi188 ceph-mgr[131698]: -- 172.21.15.188:0/2753665488 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562b1543f1e0 con 0x562b15415000 2023-07-20T02:45:59.885 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:45:59 smithi007 ceph-mon[126353]: Deploying daemon mgr.z on smithi188 2023-07-20T02:45:59.886 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:45:59 smithi158 ceph-mon[128925]: Deploying daemon mgr.z on smithi188 2023-07-20T02:46:00.063 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 20 02:45:59 smithi188 bash[131657]: 4d9639bdb8bf78a82ef54204c37430b361a59ba8edb55c91883b3d39ae851d22 2023-07-20T02:46:00.064 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 20 02:45:59 smithi188 systemd[1]: Started Ceph mgr.z for 1b04643c-26a7-11ee-9b34-001a4aab830c. 2023-07-20T02:46:00.064 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 20 02:45:59 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-z[131673]: 2023-07-20T02:45:59.889+0000 7f00adc9f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T02:46:00.064 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 20 02:45:59 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-z[131673]: 2023-07-20T02:45:59.967+0000 7f00adc9f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T02:46:00.065 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 20 02:46:00 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-z[131673]: 2023-07-20T02:46:00.055+0000 7f00adc9f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T02:46:00.324 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 20 02:46:00 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-z[131673]: 2023-07-20T02:46:00.148+0000 7f00adc9f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T02:46:00.631 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 20 02:46:00 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-z[131673]: 2023-07-20T02:46:00.316+0000 7f00adc9f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T02:46:00.631 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:00 smithi188 ceph-mon[129553]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:00.631 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:00 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:00.631 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:00 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:00.632 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:00 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:00.632 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:00 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T02:46:00.632 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:00 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T02:46:00.632 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:00 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T02:46:00.632 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:00 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:00.694 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:00 smithi007 ceph-mon[126353]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:00.694 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:00 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:00.694 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:00 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:00.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:00 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:00.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:00 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T02:46:00.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:00 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T02:46:00.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:00 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T02:46:00.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:00 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:00.825 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:00 smithi158 ceph-mon[128925]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:00.825 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:00 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:00.825 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:00 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:00.826 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:00 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:00.826 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:00 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T02:46:00.826 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:00 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T02:46:00.826 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:00 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T02:46:00.826 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:00 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:00.947 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 20 02:46:00 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-z[131673]: 2023-07-20T02:46:00.622+0000 7f00adc9f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T02:46:00.948 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 20 02:46:00 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-z[131673]: 2023-07-20T02:46:00.711+0000 7f00adc9f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T02:46:00.948 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 20 02:46:00 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-z[131673]: 2023-07-20T02:46:00.845+0000 7f00adc9f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T02:46:01.209 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 20 02:46:00 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-z[131673]: 2023-07-20T02:46:00.938+0000 7f00adc9f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T02:46:01.386 INFO:journalctl@ceph.mgr.y.smithi158.stdout:Jul 20 02:46:01 smithi158 systemd[1]: Starting Ceph mgr.y for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:46:01.505 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-20T02:46:01.506 INFO:teuthology.orchestra.run.smithi007.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-20T02:46:01.506 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10+0 records in 2023-07-20T02:46:01.506 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2023-07-20T02:46:01.506 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.040644 s, 258 MB/s 2023-07-20T02:46:01.506 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping successful for: 2023-07-20T02:46:01.709 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:01 smithi188 ceph-mon[129553]: Deploying daemon mgr.y on smithi158 2023-07-20T02:46:01.709 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 20 02:46:01 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-z[131673]: 2023-07-20T02:46:01.355+0000 7f00adc9f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T02:46:01.847 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:01 smithi158 ceph-mon[128925]: Deploying daemon mgr.y on smithi158 2023-07-20T02:46:01.847 INFO:journalctl@ceph.mgr.y.smithi158.stdout:Jul 20 02:46:01 smithi158 podman[130363]: 2023-07-20T02:46:01.911 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:01 smithi007 ceph-mon[126353]: Deploying daemon mgr.y on smithi158 2023-07-20T02:46:01.914 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi007:vg_nvme/lv_4 2023-07-20T02:46:02.136 INFO:journalctl@ceph.mgr.y.smithi158.stdout:Jul 20 02:46:01 smithi158 ceph-mgr[130403]: -- 172.21.15.158:0/808151858 <== mon.2 v2:172.21.15.158:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d396add1e0 con 0x55d396ab3000 2023-07-20T02:46:02.136 INFO:journalctl@ceph.mgr.y.smithi158.stdout:Jul 20 02:46:01 smithi158 bash[130363]: 92c4a6f2cd8c1754164cea88eeb156af01b2575c89b4ecbe8a5ae34fd0c2a587 2023-07-20T02:46:02.137 INFO:journalctl@ceph.mgr.y.smithi158.stdout:Jul 20 02:46:01 smithi158 systemd[1]: Started Ceph mgr.y for 1b04643c-26a7-11ee-9b34-001a4aab830c. 2023-07-20T02:46:02.137 INFO:journalctl@ceph.mgr.y.smithi158.stdout:Jul 20 02:46:01 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-y[130379]: 2023-07-20T02:46:01.970+0000 7f781c380200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T02:46:02.137 INFO:journalctl@ceph.mgr.y.smithi158.stdout:Jul 20 02:46:02 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-y[130379]: 2023-07-20T02:46:02.051+0000 7f781c380200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T02:46:02.209 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 20 02:46:01 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-z[131673]: 2023-07-20T02:46:01.930+0000 7f00adc9f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T02:46:02.209 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 20 02:46:02 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-z[131673]: 2023-07-20T02:46:02.101+0000 7f00adc9f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T02:46:02.427 INFO:journalctl@ceph.mgr.y.smithi158.stdout:Jul 20 02:46:02 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-y[130379]: 2023-07-20T02:46:02.139+0000 7f781c380200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T02:46:02.428 INFO:journalctl@ceph.mgr.y.smithi158.stdout:Jul 20 02:46:02 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-y[130379]: 2023-07-20T02:46:02.237+0000 7f781c380200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T02:46:02.428 INFO:journalctl@ceph.mgr.y.smithi158.stdout:Jul 20 02:46:02 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-y[130379]: 2023-07-20T02:46:02.418+0000 7f781c380200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T02:46:02.698 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:46:02.761 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:02 smithi158 ceph-mon[128925]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:02.761 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:02 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:02.761 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:02 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:02.762 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:02 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:02.762 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:02 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:02.762 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:02 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:46:02.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:02 smithi188 ceph-mon[129553]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:02.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:02 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:02.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:02 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:02.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:02 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:02.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:02 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:02.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:02 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:46:02.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:02 smithi007 ceph-mon[126353]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:02.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:02 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:02.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:02 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:02.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:02 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:02.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:02 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:02.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:02 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:46:03.099 INFO:journalctl@ceph.mgr.y.smithi158.stdout:Jul 20 02:46:02 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-y[130379]: 2023-07-20T02:46:02.751+0000 7f781c380200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T02:46:03.099 INFO:journalctl@ceph.mgr.y.smithi158.stdout:Jul 20 02:46:02 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-y[130379]: 2023-07-20T02:46:02.844+0000 7f781c380200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T02:46:03.100 INFO:journalctl@ceph.mgr.y.smithi158.stdout:Jul 20 02:46:02 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-y[130379]: 2023-07-20T02:46:02.983+0000 7f781c380200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T02:46:03.190 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 20 02:46:02 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-z[131673]: 2023-07-20T02:46:02.910+0000 7f00adc9f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T02:46:03.190 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 20 02:46:03 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-z[131673]: 2023-07-20T02:46:03.002+0000 7f00adc9f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T02:46:03.386 INFO:journalctl@ceph.mgr.y.smithi158.stdout:Jul 20 02:46:03 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-y[130379]: 2023-07-20T02:46:03.090+0000 7f781c380200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T02:46:03.459 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 20 02:46:03 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-z[131673]: 2023-07-20T02:46:03.182+0000 7f00adc9f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T02:46:03.885 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 20 02:46:03 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-z[131673]: 2023-07-20T02:46:03.477+0000 7f00adc9f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T02:46:03.886 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 20 02:46:03 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-z[131673]: 2023-07-20T02:46:03.717+0000 7f00adc9f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T02:46:03.886 INFO:journalctl@ceph.mgr.y.smithi158.stdout:Jul 20 02:46:03 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-y[130379]: 2023-07-20T02:46:03.552+0000 7f781c380200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T02:46:04.149 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 20 02:46:03 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-z[131673]: 2023-07-20T02:46:03.877+0000 7f00adc9f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T02:46:04.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.197+0000 7fd1c8892700 1 -- 172.21.15.7:0/357741226 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1c4103c30 msgr2=0x7fd1c4104090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:46:04.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.197+0000 7fd1c8892700 1 --2- 172.21.15.7:0/357741226 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1c4103c30 0x7fd1c4104090 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd1b80099c0 tx=0x7fd1b8009cd0 comp rx=0 tx=0).stop 2023-07-20T02:46:04.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.197+0000 7fd1c8892700 1 -- 172.21.15.7:0/357741226 shutdown_connections 2023-07-20T02:46:04.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.197+0000 7fd1c8892700 1 --2- 172.21.15.7:0/357741226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1c41045d0 0x7fd1c410ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:46:04.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.197+0000 7fd1c8892700 1 --2- 172.21.15.7:0/357741226 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1c4102a30 0x7fd1c4102e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:46:04.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.197+0000 7fd1c8892700 1 --2- 172.21.15.7:0/357741226 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1c4103c30 0x7fd1c4104090 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:46:04.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.197+0000 7fd1c8892700 1 -- 172.21.15.7:0/357741226 >> 172.21.15.7:0/357741226 conn(0x7fd1c40fdfd0 msgr2=0x7fd1c4100410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:46:04.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.198+0000 7fd1c8892700 1 -- 172.21.15.7:0/357741226 shutdown_connections 2023-07-20T02:46:04.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.198+0000 7fd1c8892700 1 -- 172.21.15.7:0/357741226 wait complete. 2023-07-20T02:46:04.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.199+0000 7fd1c8892700 1 Processor -- start 2023-07-20T02:46:04.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.199+0000 7fd1c8892700 1 -- start start 2023-07-20T02:46:04.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.199+0000 7fd1c8892700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1c4102a30 0x7fd1c41986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:46:04.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.199+0000 7fd1c8892700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1c4103c30 0x7fd1c4198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:46:04.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.199+0000 7fd1c8892700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1c41045d0 0x7fd1c419fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:46:04.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.199+0000 7fd1c8892700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c41a0280 con 0x7fd1c4102a30 2023-07-20T02:46:04.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.199+0000 7fd1c8892700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c41a03c0 con 0x7fd1c4103c30 2023-07-20T02:46:04.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.199+0000 7fd1c8892700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c41a0550 con 0x7fd1c41045d0 2023-07-20T02:46:04.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.199+0000 7fd1c259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1c4102a30 0x7fd1c41986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:46:04.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.200+0000 7fd1c259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1c4102a30 0x7fd1c41986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46736/0 (socket says 172.21.15.7:46736) 2023-07-20T02:46:04.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.200+0000 7fd1c259c700 1 -- 172.21.15.7:0/1098023455 learned_addr learned my addr 172.21.15.7:0/1098023455 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:46:04.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.200+0000 7fd1c1d9b700 1 --2- 172.21.15.7:0/1098023455 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1c4103c30 0x7fd1c4198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:46:04.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.200+0000 7fd1c2d9d700 1 --2- 172.21.15.7:0/1098023455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1c41045d0 0x7fd1c419fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:46:04.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.200+0000 7fd1c259c700 1 -- 172.21.15.7:0/1098023455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1c41045d0 msgr2=0x7fd1c419fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:46:04.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.200+0000 7fd1c259c700 1 --2- 172.21.15.7:0/1098023455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1c41045d0 0x7fd1c419fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:46:04.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.200+0000 7fd1c259c700 1 -- 172.21.15.7:0/1098023455 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1c4103c30 msgr2=0x7fd1c4198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:46:04.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.200+0000 7fd1c259c700 1 --2- 172.21.15.7:0/1098023455 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1c4103c30 0x7fd1c4198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:46:04.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.200+0000 7fd1c259c700 1 -- 172.21.15.7:0/1098023455 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b80095f0 con 0x7fd1c4102a30 2023-07-20T02:46:04.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.200+0000 7fd1c259c700 1 --2- 172.21.15.7:0/1098023455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1c4102a30 0x7fd1c41986f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd1b400e9a0 tx=0x7fd1b400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:46:04.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.201+0000 7fd1b37fe700 1 -- 172.21.15.7:0/1098023455 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b4009790 con 0x7fd1c4102a30 2023-07-20T02:46:04.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.201+0000 7fd1b37fe700 1 -- 172.21.15.7:0/1098023455 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd1b400c810 con 0x7fd1c4102a30 2023-07-20T02:46:04.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.201+0000 7fd1c8892700 1 -- 172.21.15.7:0/1098023455 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1c41a0830 con 0x7fd1c4102a30 2023-07-20T02:46:04.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.201+0000 7fd1b37fe700 1 -- 172.21.15.7:0/1098023455 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b4003c70 con 0x7fd1c4102a30 2023-07-20T02:46:04.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.201+0000 7fd1c8892700 1 -- 172.21.15.7:0/1098023455 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1c41a0d80 con 0x7fd1c4102a30 2023-07-20T02:46:04.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.202+0000 7fd1b37fe700 1 -- 172.21.15.7:0/1098023455 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7fd1b4021440 con 0x7fd1c4102a30 2023-07-20T02:46:04.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.202+0000 7fd1c8892700 1 -- 172.21.15.7:0/1098023455 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1c410a000 con 0x7fd1c4102a30 2023-07-20T02:46:04.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.202+0000 7fd1b37fe700 1 --2- 172.21.15.7:0/1098023455 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fd1ac0389e0 0x7fd1ac03aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:46:04.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.203+0000 7fd1b37fe700 1 -- 172.21.15.7:0/1098023455 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd1b4014070 con 0x7fd1c4102a30 2023-07-20T02:46:04.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.203+0000 7fd1c1d9b700 1 --2- 172.21.15.7:0/1098023455 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fd1ac0389e0 0x7fd1ac03aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:46:04.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.203+0000 7fd1c1d9b700 1 --2- 172.21.15.7:0/1098023455 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fd1ac0389e0 0x7fd1ac03aea0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd1b800b580 tx=0x7fd1b8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:46:04.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.208+0000 7fd1b37fe700 1 -- 172.21.15.7:0/1098023455 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd1b4020b90 con 0x7fd1c4102a30 2023-07-20T02:46:04.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:04.424+0000 7fd1c8892700 1 -- 172.21.15.7:0/1098023455 --> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fd1c4060bb0 con 0x7fd1ac0389e0 2023-07-20T02:46:04.459 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 20 02:46:04 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-z[131673]: 2023-07-20T02:46:04.140+0000 7f00adc9f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T02:46:04.530 INFO:journalctl@ceph.mgr.y.smithi158.stdout:Jul 20 02:46:04 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-y[130379]: 2023-07-20T02:46:04.140+0000 7f781c380200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T02:46:04.530 INFO:journalctl@ceph.mgr.y.smithi158.stdout:Jul 20 02:46:04 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-y[130379]: 2023-07-20T02:46:04.310+0000 7f781c380200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T02:46:04.849 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:04 smithi188 ceph-mon[129553]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:04.849 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:04 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:04.849 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:04 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:04.849 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:04 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:04.849 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:04 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:04.849 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:04 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:04.850 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:04 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:04.850 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:04 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:46:04.850 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:04 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:04.850 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:04 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T02:46:04.850 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:04 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T02:46:04.850 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:04 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:04.850 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:04 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:46:04.851 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:04 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:46:04.851 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:04 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:04.873 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:04 smithi007 ceph-mon[126353]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:04.873 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:04 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:04.873 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:04 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:04.873 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:04 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:04.873 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:04 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:04.873 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:04 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:04.874 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:04 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:04.874 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:04 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:46:04.874 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:04 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:04.874 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:04 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T02:46:04.874 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:04 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T02:46:04.874 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:04 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:04.874 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:04 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:46:04.874 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:04 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:46:04.875 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:04 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:04.886 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:04 smithi158 ceph-mon[128925]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:04.886 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:04 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:04.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:04 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:04.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:04 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:04.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:04 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:04.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:04 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:04.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:04 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:04.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:04 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:46:04.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:04 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:04.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:04 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T02:46:04.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:04 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T02:46:04.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:04 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:04.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:04 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:46:04.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:04 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:46:04.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:04 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:05.165 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 20 02:46:04 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-z[131673]: 2023-07-20T02:46:04.840+0000 7f00adc9f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T02:46:05.166 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 20 02:46:04 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-z[131673]: 2023-07-20T02:46:04.969+0000 7f00adc9f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T02:46:05.341 INFO:journalctl@ceph.mgr.y.smithi158.stdout:Jul 20 02:46:05 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-y[130379]: 2023-07-20T02:46:05.072+0000 7f781c380200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T02:46:05.341 INFO:journalctl@ceph.mgr.y.smithi158.stdout:Jul 20 02:46:05 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-y[130379]: 2023-07-20T02:46:05.161+0000 7f781c380200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T02:46:05.459 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 20 02:46:05 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-z[131673]: 2023-07-20T02:46:05.157+0000 7f00adc9f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T02:46:05.459 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 20 02:46:05 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-z[131673]: 2023-07-20T02:46:05.270+0000 7f00adc9f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T02:46:05.617 INFO:journalctl@ceph.mgr.y.smithi158.stdout:Jul 20 02:46:05 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-y[130379]: 2023-07-20T02:46:05.332+0000 7f781c380200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T02:46:05.617 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:05 smithi158 ceph-mon[128925]: Reconfiguring mgr.x (unknown last config time)... 2023-07-20T02:46:05.618 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:05 smithi158 ceph-mon[128925]: Reconfiguring daemon mgr.x on smithi007 2023-07-20T02:46:05.618 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:05 smithi158 ceph-mon[128925]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:46:05.879 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:05 smithi007 ceph-mon[126353]: Reconfiguring mgr.x (unknown last config time)... 2023-07-20T02:46:05.880 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:05 smithi007 ceph-mon[126353]: Reconfiguring daemon mgr.x on smithi007 2023-07-20T02:46:05.880 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:05 smithi007 ceph-mon[126353]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:46:05.886 INFO:journalctl@ceph.mgr.y.smithi158.stdout:Jul 20 02:46:05 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-y[130379]: 2023-07-20T02:46:05.608+0000 7f781c380200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T02:46:05.886 INFO:journalctl@ceph.mgr.y.smithi158.stdout:Jul 20 02:46:05 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-y[130379]: 2023-07-20T02:46:05.842+0000 7f781c380200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T02:46:05.959 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 20 02:46:05 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-z[131673]: 2023-07-20T02:46:05.483+0000 7f00adc9f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T02:46:06.000 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:05 smithi188 ceph-mon[129553]: Reconfiguring mgr.x (unknown last config time)... 2023-07-20T02:46:06.000 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:05 smithi188 ceph-mon[129553]: Reconfiguring daemon mgr.x on smithi007 2023-07-20T02:46:06.000 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:05 smithi188 ceph-mon[129553]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:46:06.253 INFO:journalctl@ceph.mgr.y.smithi158.stdout:Jul 20 02:46:05 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-y[130379]: 2023-07-20T02:46:05.987+0000 7f781c380200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T02:46:06.593 INFO:journalctl@ceph.mgr.y.smithi158.stdout:Jul 20 02:46:06 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-y[130379]: 2023-07-20T02:46:06.245+0000 7f781c380200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T02:46:06.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:06.622+0000 7fd1b37fe700 1 -- 172.21.15.7:0/1098023455 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mgrmap(e 14) v1 ==== 90598+0+0 (secure 0 0 0) 0x7fd1b40203a0 con 0x7fd1c4102a30 2023-07-20T02:46:06.710 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 20 02:46:06 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-z[131673]: 2023-07-20T02:46:06.343+0000 7f00adc9f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T02:46:06.710 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:06 smithi188 ceph-mon[129553]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:06.711 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:06 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:06.711 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:06 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:06.711 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:06 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:46:06.711 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:06 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:06.711 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:06 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:46:06.711 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:06 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:06.711 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:06 smithi188 ceph-mon[129553]: Standby manager daemon z started 2023-07-20T02:46:06.711 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:06 smithi188 ceph-mon[129553]: from='mgr.? 172.21.15.188:0/2074307845' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-20T02:46:06.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:06 smithi188 ceph-mon[129553]: from='mgr.? 172.21.15.188:0/2074307845' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T02:46:06.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:06 smithi188 ceph-mon[129553]: from='mgr.? 172.21.15.188:0/2074307845' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-20T02:46:06.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:06 smithi188 ceph-mon[129553]: from='mgr.? 172.21.15.188:0/2074307845' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T02:46:06.814 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:06 smithi007 ceph-mon[126353]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:06.815 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:06 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:06.815 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:06 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:06.815 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:06 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:46:06.815 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:06 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:06.815 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:06 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:46:06.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:06 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:06.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:06 smithi007 ceph-mon[126353]: Standby manager daemon z started 2023-07-20T02:46:06.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:06 smithi007 ceph-mon[126353]: from='mgr.? 172.21.15.188:0/2074307845' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-20T02:46:06.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:06 smithi007 ceph-mon[126353]: from='mgr.? 172.21.15.188:0/2074307845' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T02:46:06.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:06 smithi007 ceph-mon[126353]: from='mgr.? 172.21.15.188:0/2074307845' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-20T02:46:06.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:06 smithi007 ceph-mon[126353]: from='mgr.? 172.21.15.188:0/2074307845' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T02:46:06.886 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:06 smithi158 ceph-mon[128925]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:06.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:06 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:06.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:06 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:06.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:06 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:46:06.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:06 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:06.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:06 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:46:06.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:06 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:06.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:06 smithi158 ceph-mon[128925]: Standby manager daemon z started 2023-07-20T02:46:06.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:06 smithi158 ceph-mon[128925]: from='mgr.? 172.21.15.188:0/2074307845' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-20T02:46:06.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:06 smithi158 ceph-mon[128925]: from='mgr.? 172.21.15.188:0/2074307845' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T02:46:06.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:06 smithi158 ceph-mon[128925]: from='mgr.? 172.21.15.188:0/2074307845' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-20T02:46:06.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:06 smithi158 ceph-mon[128925]: from='mgr.? 172.21.15.188:0/2074307845' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T02:46:07.271 INFO:journalctl@ceph.mgr.y.smithi158.stdout:Jul 20 02:46:06 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-y[130379]: 2023-07-20T02:46:06.945+0000 7f781c380200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T02:46:07.272 INFO:journalctl@ceph.mgr.y.smithi158.stdout:Jul 20 02:46:07 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-y[130379]: 2023-07-20T02:46:07.074+0000 7f781c380200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T02:46:07.598 INFO:journalctl@ceph.mgr.y.smithi158.stdout:Jul 20 02:46:07 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-y[130379]: 2023-07-20T02:46:07.262+0000 7f781c380200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T02:46:07.598 INFO:journalctl@ceph.mgr.y.smithi158.stdout:Jul 20 02:46:07 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-y[130379]: 2023-07-20T02:46:07.375+0000 7f781c380200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T02:46:07.886 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:07 smithi158 ceph-mon[128925]: mgrmap e14: x(active, since 73s), standbys: z 2023-07-20T02:46:07.886 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:07 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-20T02:46:07.887 INFO:journalctl@ceph.mgr.y.smithi158.stdout:Jul 20 02:46:07 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-y[130379]: 2023-07-20T02:46:07.589+0000 7f781c380200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T02:46:07.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:07 smithi007 ceph-mon[126353]: mgrmap e14: x(active, since 73s), standbys: z 2023-07-20T02:46:07.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:07 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-20T02:46:07.959 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:07 smithi188 ceph-mon[129553]: mgrmap e14: x(active, since 73s), standbys: z 2023-07-20T02:46:07.959 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:07 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-20T02:46:08.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:08.814+0000 7fd1b37fe700 1 -- 172.21.15.7:0/1098023455 <== mon.0 v2:172.21.15.7:3300/0 8 ==== mgrmap(e 15) v1 ==== 135658+0+0 (secure 0 0 0) 0x7fd1b407d4c0 con 0x7fd1c4102a30 2023-07-20T02:46:08.891 INFO:journalctl@ceph.mgr.y.smithi158.stdout:Jul 20 02:46:08 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-y[130379]: 2023-07-20T02:46:08.447+0000 7f781c380200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T02:46:08.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:08 smithi158 ceph-mon[128925]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:08.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:08 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/511538286' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad6c6339-6e23-4c21-9fde-2dbecdafdb17"}]: dispatch 2023-07-20T02:46:08.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:08 smithi158 ceph-mon[128925]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad6c6339-6e23-4c21-9fde-2dbecdafdb17"}]: dispatch 2023-07-20T02:46:08.892 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:08 smithi158 ceph-mon[128925]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad6c6339-6e23-4c21-9fde-2dbecdafdb17"}]': finished 2023-07-20T02:46:08.892 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:08 smithi158 ceph-mon[128925]: osdmap e5: 1 total, 0 up, 1 in 2023-07-20T02:46:08.892 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:08 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T02:46:08.892 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:08 smithi158 ceph-mon[128925]: Standby manager daemon y started 2023-07-20T02:46:08.892 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:08 smithi158 ceph-mon[128925]: from='mgr.? 172.21.15.158:0/1626434299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-20T02:46:08.892 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:08 smithi158 ceph-mon[128925]: from='mgr.? 172.21.15.158:0/1626434299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T02:46:08.892 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:08 smithi158 ceph-mon[128925]: from='mgr.? 172.21.15.158:0/1626434299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-20T02:46:08.892 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:08 smithi158 ceph-mon[128925]: from='mgr.? 172.21.15.158:0/1626434299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T02:46:08.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:08 smithi007 ceph-mon[126353]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:08.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:08 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/511538286' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad6c6339-6e23-4c21-9fde-2dbecdafdb17"}]: dispatch 2023-07-20T02:46:08.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:08 smithi007 ceph-mon[126353]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad6c6339-6e23-4c21-9fde-2dbecdafdb17"}]: dispatch 2023-07-20T02:46:08.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:08 smithi007 ceph-mon[126353]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad6c6339-6e23-4c21-9fde-2dbecdafdb17"}]': finished 2023-07-20T02:46:08.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:08 smithi007 ceph-mon[126353]: osdmap e5: 1 total, 0 up, 1 in 2023-07-20T02:46:08.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:08 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T02:46:08.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:08 smithi007 ceph-mon[126353]: Standby manager daemon y started 2023-07-20T02:46:08.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:08 smithi007 ceph-mon[126353]: from='mgr.? 172.21.15.158:0/1626434299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-20T02:46:08.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:08 smithi007 ceph-mon[126353]: from='mgr.? 172.21.15.158:0/1626434299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T02:46:08.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:08 smithi007 ceph-mon[126353]: from='mgr.? 172.21.15.158:0/1626434299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-20T02:46:08.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:08 smithi007 ceph-mon[126353]: from='mgr.? 172.21.15.158:0/1626434299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T02:46:08.959 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:08 smithi188 ceph-mon[129553]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:08.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:08 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/511538286' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad6c6339-6e23-4c21-9fde-2dbecdafdb17"}]: dispatch 2023-07-20T02:46:08.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:08 smithi188 ceph-mon[129553]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad6c6339-6e23-4c21-9fde-2dbecdafdb17"}]: dispatch 2023-07-20T02:46:08.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:08 smithi188 ceph-mon[129553]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad6c6339-6e23-4c21-9fde-2dbecdafdb17"}]': finished 2023-07-20T02:46:08.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:08 smithi188 ceph-mon[129553]: osdmap e5: 1 total, 0 up, 1 in 2023-07-20T02:46:08.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:08 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T02:46:08.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:08 smithi188 ceph-mon[129553]: Standby manager daemon y started 2023-07-20T02:46:08.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:08 smithi188 ceph-mon[129553]: from='mgr.? 172.21.15.158:0/1626434299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-20T02:46:08.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:08 smithi188 ceph-mon[129553]: from='mgr.? 172.21.15.158:0/1626434299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T02:46:08.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:08 smithi188 ceph-mon[129553]: from='mgr.? 172.21.15.158:0/1626434299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-20T02:46:08.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:08 smithi188 ceph-mon[129553]: from='mgr.? 172.21.15.158:0/1626434299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T02:46:09.886 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:09 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/2560834672' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:46:09.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:09 smithi158 ceph-mon[128925]: mgrmap e15: x(active, since 75s), standbys: z, y 2023-07-20T02:46:09.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:09 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-20T02:46:09.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:09 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/2560834672' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:46:09.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:09 smithi007 ceph-mon[126353]: mgrmap e15: x(active, since 75s), standbys: z, y 2023-07-20T02:46:09.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:09 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-20T02:46:09.959 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:09 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/2560834672' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:46:09.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:09 smithi188 ceph-mon[129553]: mgrmap e15: x(active, since 75s), standbys: z, y 2023-07-20T02:46:09.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:09 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-20T02:46:10.886 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:10 smithi158 ceph-mon[128925]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:10.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:10 smithi007 ceph-mon[126353]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:10.959 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:10 smithi188 ceph-mon[129553]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:11.897 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:11 smithi007 ceph-mon[126353]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:11.959 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:11 smithi188 ceph-mon[129553]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:12.136 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:11 smithi158 ceph-mon[128925]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:14.694 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:14 smithi007 ceph-mon[126353]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:14.886 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:14 smithi158 ceph-mon[128925]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:14.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:14 smithi188 ceph-mon[129553]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:16.852 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:16 smithi007 ceph-mon[126353]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:16.853 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:16 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-20T02:46:16.853 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:16 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:16.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:16 smithi158 ceph-mon[128925]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:16.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:16 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-20T02:46:16.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:16 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:16.959 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:16 smithi188 ceph-mon[129553]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:16.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:16 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-20T02:46:16.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:16 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:17.886 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:17 smithi158 ceph-mon[128925]: Deploying daemon osd.0 on smithi007 2023-07-20T02:46:17.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:17 smithi007 ceph-mon[126353]: Deploying daemon osd.0 on smithi007 2023-07-20T02:46:17.959 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:17 smithi188 ceph-mon[129553]: Deploying daemon osd.0 on smithi007 2023-07-20T02:46:18.694 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:18 smithi007 ceph-mon[126353]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:18.886 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:18 smithi158 ceph-mon[128925]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:18.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:18 smithi188 ceph-mon[129553]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:20.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:20 smithi007 ceph-mon[126353]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:20.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:20 smithi158 ceph-mon[128925]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:20.959 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:20 smithi188 ceph-mon[129553]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:22.886 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:22 smithi158 ceph-mon[128925]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:22.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:22 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:22.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:22 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:22.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:22 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:46:22.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:22 smithi007 ceph-mon[126353]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:22.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:22 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:22.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:22 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:22.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:22 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:46:22.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:22 smithi188 ceph-mon[129553]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:22.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:22 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:22.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:22 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:22.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:22 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:46:24.694 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:24 smithi007 ceph-mon[126353]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:24.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:24 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:24.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:24 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:24.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:24 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:24.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:24 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:46:24.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:24 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:24.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:24 smithi158 ceph-mon[128925]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:24.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:24 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:24.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:24 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:24.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:24 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:24.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:24 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:46:24.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:24 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:24.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:24 smithi188 ceph-mon[129553]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:24.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:24 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:24.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:24 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:24.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:24 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:24.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:24 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:46:24.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:24 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:25.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:25.786+0000 7fd1b37fe700 1 -- 172.21.15.7:0/1098023455 <== mgr.14152 v2:172.21.15.7:6800/1354542137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd1c4060bb0 con 0x7fd1ac0389e0 2023-07-20T02:46:25.797 INFO:teuthology.orchestra.run.smithi007.stdout:Created osd(s) 0 on host 'smithi007' 2023-07-20T02:46:25.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:25.790+0000 7fd1c8892700 1 -- 172.21.15.7:0/1098023455 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fd1ac0389e0 msgr2=0x7fd1ac03aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:46:25.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:25.790+0000 7fd1c8892700 1 --2- 172.21.15.7:0/1098023455 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fd1ac0389e0 0x7fd1ac03aea0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd1b800b580 tx=0x7fd1b8005fb0 comp rx=0 tx=0).stop 2023-07-20T02:46:25.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:25.790+0000 7fd1c8892700 1 -- 172.21.15.7:0/1098023455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1c4102a30 msgr2=0x7fd1c41986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:46:25.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:25.790+0000 7fd1c8892700 1 --2- 172.21.15.7:0/1098023455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1c4102a30 0x7fd1c41986f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd1b400e9a0 tx=0x7fd1b400ecb0 comp rx=0 tx=0).stop 2023-07-20T02:46:25.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:25.790+0000 7fd1c8892700 1 -- 172.21.15.7:0/1098023455 shutdown_connections 2023-07-20T02:46:25.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:25.790+0000 7fd1c8892700 1 --2- 172.21.15.7:0/1098023455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1c41045d0 0x7fd1c419fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:46:25.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:25.790+0000 7fd1c8892700 1 --2- 172.21.15.7:0/1098023455 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fd1ac0389e0 0x7fd1ac03aea0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:46:25.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:25.790+0000 7fd1c8892700 1 --2- 172.21.15.7:0/1098023455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1c4102a30 0x7fd1c41986f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:46:25.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:25.790+0000 7fd1c8892700 1 --2- 172.21.15.7:0/1098023455 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd1c4103c30 0x7fd1c4198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:46:25.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:25.790+0000 7fd1c8892700 1 -- 172.21.15.7:0/1098023455 >> 172.21.15.7:0/1098023455 conn(0x7fd1c40fdfd0 msgr2=0x7fd1c4107140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:46:25.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:25.791+0000 7fd1c8892700 1 -- 172.21.15.7:0/1098023455 shutdown_connections 2023-07-20T02:46:25.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:46:25.791+0000 7fd1c8892700 1 -- 172.21.15.7:0/1098023455 wait complete. 2023-07-20T02:46:26.324 DEBUG:teuthology.orchestra.run.smithi007:osd.0> sudo journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.0.service 2023-07-20T02:46:26.330 INFO:tasks.cephadm:Deploying osd.1 on smithi158 with /dev/vg_nvme/lv_4... 2023-07-20T02:46:26.330 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-20T02:46:26.406 INFO:journalctl@ceph.osd.0.smithi007.stdout:-- Logs begin at Thu 2023-07-20 02:22:32 UTC. -- 2023-07-20T02:46:26.637 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:26 smithi158 ceph-mon[128925]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:26.637 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:26 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:26.637 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:26 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:26.638 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:26 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:46:26.638 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:26 smithi158 ceph-mon[128925]: from='osd.0 [v2:172.21.15.7:6802/3322992091,v1:172.21.15.7:6803/3322992091]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-20T02:46:26.638 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:26 smithi158 ceph-mon[128925]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-20T02:46:26.694 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:26 smithi007 ceph-mon[126353]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:26.694 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:26 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:26.694 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:26 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:26.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:26 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:46:26.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:26 smithi007 ceph-mon[126353]: from='osd.0 [v2:172.21.15.7:6802/3322992091,v1:172.21.15.7:6803/3322992091]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-20T02:46:26.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:26 smithi007 ceph-mon[126353]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-20T02:46:26.892 INFO:teuthology.orchestra.run.smithi158.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T02:46:26.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:26 smithi188 ceph-mon[129553]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:26.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:26 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:26.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:26 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:26.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:26 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:46:26.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:26 smithi188 ceph-mon[129553]: from='osd.0 [v2:172.21.15.7:6802/3322992091,v1:172.21.15.7:6803/3322992091]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-20T02:46:26.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:26 smithi188 ceph-mon[129553]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-20T02:46:27.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:27 smithi007 ceph-mon[126353]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-20T02:46:27.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:27 smithi007 ceph-mon[126353]: osdmap e6: 1 total, 0 up, 1 in 2023-07-20T02:46:27.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:27 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T02:46:27.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:27 smithi007 ceph-mon[126353]: from='osd.0 [v2:172.21.15.7:6802/3322992091,v1:172.21.15.7:6803/3322992091]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-07-20T02:46:27.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:27 smithi007 ceph-mon[126353]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-07-20T02:46:27.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:27 smithi007 ceph-mon[126353]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:27.946 INFO:journalctl@ceph.osd.0.smithi007.stdout:Jul 20 02:46:27 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-0[139430]: 2023-07-20T02:46:27.764+0000 7fa575b43700 -1 osd.0 0 waiting for initial osdmap 2023-07-20T02:46:27.946 INFO:journalctl@ceph.osd.0.smithi007.stdout:Jul 20 02:46:27 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-0[139430]: 2023-07-20T02:46:27.769+0000 7fa56d130700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T02:46:28.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:27 smithi158 ceph-mon[128925]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-20T02:46:28.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:27 smithi158 ceph-mon[128925]: osdmap e6: 1 total, 0 up, 1 in 2023-07-20T02:46:28.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:27 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T02:46:28.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:27 smithi158 ceph-mon[128925]: from='osd.0 [v2:172.21.15.7:6802/3322992091,v1:172.21.15.7:6803/3322992091]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-07-20T02:46:28.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:27 smithi158 ceph-mon[128925]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-07-20T02:46:28.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:27 smithi158 ceph-mon[128925]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:28.209 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:27 smithi188 ceph-mon[129553]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-20T02:46:28.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:27 smithi188 ceph-mon[129553]: osdmap e6: 1 total, 0 up, 1 in 2023-07-20T02:46:28.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:27 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T02:46:28.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:27 smithi188 ceph-mon[129553]: from='osd.0 [v2:172.21.15.7:6802/3322992091,v1:172.21.15.7:6803/3322992091]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-07-20T02:46:28.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:27 smithi188 ceph-mon[129553]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-07-20T02:46:28.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:27 smithi188 ceph-mon[129553]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:28.285 INFO:teuthology.orchestra.run.smithi158.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-20T02:46:28.285 INFO:teuthology.orchestra.run.smithi158.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-20T02:46:28.285 INFO:teuthology.orchestra.run.smithi158.stderr: stderr: 10+0 records in 2023-07-20T02:46:28.285 INFO:teuthology.orchestra.run.smithi158.stderr:10+0 records out 2023-07-20T02:46:28.285 INFO:teuthology.orchestra.run.smithi158.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00825411 s, 1.3 GB/s 2023-07-20T02:46:28.286 INFO:teuthology.orchestra.run.smithi158.stderr:--> Zapping successful for: 2023-07-20T02:46:28.630 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi158:vg_nvme/lv_4 2023-07-20T02:46:28.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:28 smithi158 ceph-mon[128925]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2023-07-20T02:46:28.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:28 smithi158 ceph-mon[128925]: osdmap e7: 1 total, 0 up, 1 in 2023-07-20T02:46:28.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:28 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T02:46:28.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:28 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T02:46:29.139 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:28 smithi007 ceph-mon[126353]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2023-07-20T02:46:29.140 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:28 smithi007 ceph-mon[126353]: osdmap e7: 1 total, 0 up, 1 in 2023-07-20T02:46:29.140 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:28 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T02:46:29.140 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:28 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T02:46:29.187 INFO:teuthology.orchestra.run.smithi158.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T02:46:29.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:28 smithi188 ceph-mon[129553]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2023-07-20T02:46:29.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:28 smithi188 ceph-mon[129553]: osdmap e7: 1 total, 0 up, 1 in 2023-07-20T02:46:29.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:28 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T02:46:29.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:28 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T02:46:30.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:29 smithi158 ceph-mon[128925]: purged_snaps scrub starts 2023-07-20T02:46:30.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:29 smithi158 ceph-mon[128925]: purged_snaps scrub ok 2023-07-20T02:46:30.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:29 smithi158 ceph-mon[128925]: osd.0 [v2:172.21.15.7:6802/3322992091,v1:172.21.15.7:6803/3322992091] boot 2023-07-20T02:46:30.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:29 smithi158 ceph-mon[128925]: osdmap e8: 1 total, 1 up, 1 in 2023-07-20T02:46:30.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:29 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T02:46:30.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:29 smithi158 ceph-mon[128925]: Detected new or changed devices on smithi007 2023-07-20T02:46:30.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:29 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:30.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:29 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:30.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:29 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:46:30.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:29 smithi158 ceph-mon[128925]: Adjusting osd_memory_target on smithi007 to 17272M 2023-07-20T02:46:30.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:29 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:30.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:29 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:30.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:29 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:46:30.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:29 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:30.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:29 smithi158 ceph-mon[128925]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:30.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:29 smithi007 ceph-mon[126353]: purged_snaps scrub starts 2023-07-20T02:46:30.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:29 smithi007 ceph-mon[126353]: purged_snaps scrub ok 2023-07-20T02:46:30.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:29 smithi007 ceph-mon[126353]: osd.0 [v2:172.21.15.7:6802/3322992091,v1:172.21.15.7:6803/3322992091] boot 2023-07-20T02:46:30.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:29 smithi007 ceph-mon[126353]: osdmap e8: 1 total, 1 up, 1 in 2023-07-20T02:46:30.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:29 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T02:46:30.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:29 smithi007 ceph-mon[126353]: Detected new or changed devices on smithi007 2023-07-20T02:46:30.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:29 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:30.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:29 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:30.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:29 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:46:30.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:29 smithi007 ceph-mon[126353]: Adjusting osd_memory_target on smithi007 to 17272M 2023-07-20T02:46:30.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:29 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:30.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:29 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:30.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:29 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:46:30.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:29 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:30.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:29 smithi007 ceph-mon[126353]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:30.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:29 smithi188 ceph-mon[129553]: purged_snaps scrub starts 2023-07-20T02:46:30.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:29 smithi188 ceph-mon[129553]: purged_snaps scrub ok 2023-07-20T02:46:30.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:29 smithi188 ceph-mon[129553]: osd.0 [v2:172.21.15.7:6802/3322992091,v1:172.21.15.7:6803/3322992091] boot 2023-07-20T02:46:30.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:29 smithi188 ceph-mon[129553]: osdmap e8: 1 total, 1 up, 1 in 2023-07-20T02:46:30.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:29 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T02:46:30.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:29 smithi188 ceph-mon[129553]: Detected new or changed devices on smithi007 2023-07-20T02:46:30.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:29 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:30.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:29 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:30.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:29 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:46:30.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:29 smithi188 ceph-mon[129553]: Adjusting osd_memory_target on smithi007 to 17272M 2023-07-20T02:46:30.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:29 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:30.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:29 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:30.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:29 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:46:30.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:29 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:30.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:29 smithi188 ceph-mon[129553]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T02:46:30.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.365+0000 7f6059f7a700 1 -- 172.21.15.158:0/574422741 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60540ff9d0 msgr2=0x7f60540ffe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:46:30.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.365+0000 7f6059f7a700 1 --2- 172.21.15.158:0/574422741 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60540ff9d0 0x7f60540ffe30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6044009990 tx=0x7f6044009ca0 comp rx=0 tx=0).stop 2023-07-20T02:46:30.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.365+0000 7f6059f7a700 1 -- 172.21.15.158:0/574422741 shutdown_connections 2023-07-20T02:46:30.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.365+0000 7f6059f7a700 1 --2- 172.21.15.158:0/574422741 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6054100370 0x7f6054107c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:46:30.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.365+0000 7f6059f7a700 1 --2- 172.21.15.158:0/574422741 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6054100c90 0x7f60541010b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:46:30.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.365+0000 7f6059f7a700 1 --2- 172.21.15.158:0/574422741 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60540ff9d0 0x7f60540ffe30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:46:30.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.365+0000 7f6059f7a700 1 -- 172.21.15.158:0/574422741 >> 172.21.15.158:0/574422741 conn(0x7f60540fa420 msgr2=0x7f60540fc860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:46:30.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.365+0000 7f6059f7a700 1 -- 172.21.15.158:0/574422741 shutdown_connections 2023-07-20T02:46:30.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.365+0000 7f6059f7a700 1 -- 172.21.15.158:0/574422741 wait complete. 2023-07-20T02:46:30.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.366+0000 7f6059f7a700 1 Processor -- start 2023-07-20T02:46:30.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.366+0000 7f6059f7a700 1 -- start start 2023-07-20T02:46:30.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.366+0000 7f6059f7a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60540ff9d0 0x7f60540ff420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:46:30.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.366+0000 7f6059f7a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6054100370 0x7f605419b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:46:30.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.366+0000 7f6059f7a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6054100c90 0x7f605419d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:46:30.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.366+0000 7f6059f7a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605419dfc0 con 0x7f6054100c90 2023-07-20T02:46:30.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.366+0000 7f6059f7a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605419e130 con 0x7f60540ff9d0 2023-07-20T02:46:30.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.366+0000 7f6059f7a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605419e2f0 con 0x7f6054100370 2023-07-20T02:46:30.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.366+0000 7f60537fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60540ff9d0 0x7f60540ff420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:46:30.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.366+0000 7f60537fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60540ff9d0 0x7f60540ff420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58184/0 (socket says 172.21.15.158:58184) 2023-07-20T02:46:30.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.366+0000 7f60537fe700 1 -- 172.21.15.158:0/3506180596 learned_addr learned my addr 172.21.15.158:0/3506180596 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-07-20T02:46:30.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.367+0000 7f6052ffd700 1 --2- 172.21.15.158:0/3506180596 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6054100370 0x7f605419b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:46:30.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.367+0000 7f60537fe700 1 -- 172.21.15.158:0/3506180596 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6054100370 msgr2=0x7f605419b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:46:30.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.367+0000 7f60537fe700 1 --2- 172.21.15.158:0/3506180596 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6054100370 0x7f605419b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:46:30.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.367+0000 7f60537fe700 1 -- 172.21.15.158:0/3506180596 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6054100c90 msgr2=0x7f605419d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:46:30.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.367+0000 7f60537fe700 1 --2- 172.21.15.158:0/3506180596 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6054100c90 0x7f605419d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:46:30.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.367+0000 7f60537fe700 1 -- 172.21.15.158:0/3506180596 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60440095f0 con 0x7f60540ff9d0 2023-07-20T02:46:30.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.367+0000 7f60537fe700 1 --2- 172.21.15.158:0/3506180596 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60540ff9d0 0x7f60540ff420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f604800d980 tx=0x7f604800dd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:46:30.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.367+0000 7f6050ff9700 1 -- 172.21.15.158:0/3506180596 <== mon.2 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604800de60 con 0x7f60540ff9d0 2023-07-20T02:46:30.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.367+0000 7f6050ff9700 1 -- 172.21.15.158:0/3506180596 <== mon.2 v2:172.21.15.158:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f60480044e0 con 0x7f60540ff9d0 2023-07-20T02:46:30.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.367+0000 7f6059f7a700 1 -- 172.21.15.158:0/3506180596 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f605419e5d0 con 0x7f60540ff9d0 2023-07-20T02:46:30.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.367+0000 7f6050ff9700 1 -- 172.21.15.158:0/3506180596 <== mon.2 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6048003be0 con 0x7f60540ff9d0 2023-07-20T02:46:30.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.367+0000 7f6059f7a700 1 -- 172.21.15.158:0/3506180596 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f605419eb20 con 0x7f60540ff9d0 2023-07-20T02:46:30.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.368+0000 7f6059f7a700 1 -- 172.21.15.158:0/3506180596 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f605410aef0 con 0x7f60540ff9d0 2023-07-20T02:46:30.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.369+0000 7f6050ff9700 1 -- 172.21.15.158:0/3506180596 <== mon.2 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 15) v1 ==== 135658+0+0 (secure 0 0 0) 0x7f6048009bb0 con 0x7f60540ff9d0 2023-07-20T02:46:30.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.369+0000 7f6050ff9700 1 --2- 172.21.15.158:0/3506180596 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f603c0a11c0 0x7f603c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:46:30.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.370+0000 7f6050ff9700 1 -- 172.21.15.158:0/3506180596 <== mon.2 v2:172.21.15.158:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f60480cac30 con 0x7f60540ff9d0 2023-07-20T02:46:30.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.370+0000 7f6052ffd700 1 --2- 172.21.15.158:0/3506180596 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f603c0a11c0 0x7f603c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:46:30.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.371+0000 7f6052ffd700 1 --2- 172.21.15.158:0/3506180596 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f603c0a11c0 0x7f603c0a3680 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6044006010 tx=0x7f604400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:46:30.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.374+0000 7f6050ff9700 1 -- 172.21.15.158:0/3506180596 <== mon.2 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6048095d50 con 0x7f60540ff9d0 2023-07-20T02:46:30.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:30.565+0000 7f6059f7a700 1 -- 172.21.15.158:0/3506180596 --> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi158:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f6054060bb0 con 0x7f603c0a11c0 2023-07-20T02:46:31.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:30 smithi158 ceph-mon[128925]: osdmap e9: 1 total, 1 up, 1 in 2023-07-20T02:46:31.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:30 smithi158 ceph-mon[128925]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi158:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:46:31.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:30 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:46:31.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:30 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:46:31.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:30 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:31.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:30 smithi007 ceph-mon[126353]: osdmap e9: 1 total, 1 up, 1 in 2023-07-20T02:46:31.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:30 smithi007 ceph-mon[126353]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi158:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:46:31.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:30 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:46:31.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:30 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:46:31.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:30 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:31.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:30 smithi188 ceph-mon[129553]: osdmap e9: 1 total, 1 up, 1 in 2023-07-20T02:46:31.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:30 smithi188 ceph-mon[129553]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi158:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:46:31.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:30 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:46:31.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:30 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:46:31.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:30 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:32.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:31 smithi158 ceph-mon[128925]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:46:32.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:31 smithi007 ceph-mon[126353]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:46:32.209 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:31 smithi188 ceph-mon[129553]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:46:33.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:32 smithi158 ceph-mon[128925]: from='client.? 172.21.15.158:0/3124856076' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cad207b3-a1ea-4f07-bd94-9b11fd573a69"}]: dispatch 2023-07-20T02:46:33.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:32 smithi158 ceph-mon[128925]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cad207b3-a1ea-4f07-bd94-9b11fd573a69"}]: dispatch 2023-07-20T02:46:33.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:32 smithi158 ceph-mon[128925]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cad207b3-a1ea-4f07-bd94-9b11fd573a69"}]': finished 2023-07-20T02:46:33.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:32 smithi158 ceph-mon[128925]: osdmap e10: 2 total, 1 up, 2 in 2023-07-20T02:46:33.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:32 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T02:46:33.444 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:32 smithi007 ceph-mon[126353]: from='client.? 172.21.15.158:0/3124856076' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cad207b3-a1ea-4f07-bd94-9b11fd573a69"}]: dispatch 2023-07-20T02:46:33.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:32 smithi007 ceph-mon[126353]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cad207b3-a1ea-4f07-bd94-9b11fd573a69"}]: dispatch 2023-07-20T02:46:33.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:32 smithi007 ceph-mon[126353]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cad207b3-a1ea-4f07-bd94-9b11fd573a69"}]': finished 2023-07-20T02:46:33.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:32 smithi007 ceph-mon[126353]: osdmap e10: 2 total, 1 up, 2 in 2023-07-20T02:46:33.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:32 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T02:46:33.460 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:32 smithi188 ceph-mon[129553]: from='client.? 172.21.15.158:0/3124856076' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cad207b3-a1ea-4f07-bd94-9b11fd573a69"}]: dispatch 2023-07-20T02:46:33.460 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:32 smithi188 ceph-mon[129553]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cad207b3-a1ea-4f07-bd94-9b11fd573a69"}]: dispatch 2023-07-20T02:46:33.460 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:32 smithi188 ceph-mon[129553]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cad207b3-a1ea-4f07-bd94-9b11fd573a69"}]': finished 2023-07-20T02:46:33.460 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:32 smithi188 ceph-mon[129553]: osdmap e10: 2 total, 1 up, 2 in 2023-07-20T02:46:33.460 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:32 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T02:46:34.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:33 smithi158 ceph-mon[128925]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:46:34.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:33 smithi158 ceph-mon[128925]: from='client.? 172.21.15.158:0/1900257272' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:46:34.444 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:33 smithi007 ceph-mon[126353]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:46:34.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:33 smithi007 ceph-mon[126353]: from='client.? 172.21.15.158:0/1900257272' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:46:34.460 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:33 smithi188 ceph-mon[129553]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:46:34.460 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:33 smithi188 ceph-mon[129553]: from='client.? 172.21.15.158:0/1900257272' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:46:36.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:36 smithi158 ceph-mon[128925]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:46:36.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:36 smithi007 ceph-mon[126353]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:46:36.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:36 smithi188 ceph-mon[129553]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:46:38.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:37 smithi158 ceph-mon[128925]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:46:38.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:37 smithi007 ceph-mon[126353]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:46:38.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:37 smithi188 ceph-mon[129553]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:46:39.637 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:39 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-20T02:46:39.637 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:39 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:39.694 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:39 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-20T02:46:39.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:39 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:39.710 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:39 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-20T02:46:39.710 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:39 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:40.637 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:40 smithi158 ceph-mon[128925]: Deploying daemon osd.1 on smithi158 2023-07-20T02:46:40.637 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:40 smithi158 ceph-mon[128925]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:46:40.694 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:40 smithi007 ceph-mon[126353]: Deploying daemon osd.1 on smithi158 2023-07-20T02:46:40.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:40 smithi007 ceph-mon[126353]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:46:40.710 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:40 smithi188 ceph-mon[129553]: Deploying daemon osd.1 on smithi158 2023-07-20T02:46:40.710 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:40 smithi188 ceph-mon[129553]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:46:42.637 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:42 smithi158 ceph-mon[128925]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:46:42.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:42 smithi007 ceph-mon[126353]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:46:42.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:42 smithi188 ceph-mon[129553]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:46:44.800 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:44 smithi158 ceph-mon[128925]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:46:44.801 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:44 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:44.801 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:44 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:46:44.801 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:44 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:44.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:44 smithi007 ceph-mon[126353]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:46:44.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:44 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:44.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:44 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:46:44.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:44 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:44.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:44 smithi188 ceph-mon[129553]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:46:44.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:44 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:44.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:44 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:46:44.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:44 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:46.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:46.381+0000 7f6050ff9700 1 -- 172.21.15.158:0/3506180596 <== mgr.14152 v2:172.21.15.7:6800/1354542137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6054060bb0 con 0x7f603c0a11c0 2023-07-20T02:46:46.391 INFO:teuthology.orchestra.run.smithi158.stdout:Created osd(s) 1 on host 'smithi158' 2023-07-20T02:46:46.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:46.383+0000 7f6059f7a700 1 -- 172.21.15.158:0/3506180596 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f603c0a11c0 msgr2=0x7f603c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:46:46.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:46.383+0000 7f6059f7a700 1 --2- 172.21.15.158:0/3506180596 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f603c0a11c0 0x7f603c0a3680 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6044006010 tx=0x7f604400b540 comp rx=0 tx=0).stop 2023-07-20T02:46:46.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:46.383+0000 7f6059f7a700 1 -- 172.21.15.158:0/3506180596 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60540ff9d0 msgr2=0x7f60540ff420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:46:46.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:46.383+0000 7f6059f7a700 1 --2- 172.21.15.158:0/3506180596 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60540ff9d0 0x7f60540ff420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f604800d980 tx=0x7f604800dd40 comp rx=0 tx=0).stop 2023-07-20T02:46:46.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:46.383+0000 7f6059f7a700 1 -- 172.21.15.158:0/3506180596 shutdown_connections 2023-07-20T02:46:46.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:46.383+0000 7f6059f7a700 1 --2- 172.21.15.158:0/3506180596 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f603c0a11c0 0x7f603c0a3680 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:46:46.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:46.383+0000 7f6059f7a700 1 --2- 172.21.15.158:0/3506180596 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6054100c90 0x7f605419d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:46:46.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:46.383+0000 7f6059f7a700 1 --2- 172.21.15.158:0/3506180596 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f60540ff9d0 0x7f60540ff420 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:46:46.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:46.383+0000 7f6059f7a700 1 --2- 172.21.15.158:0/3506180596 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6054100370 0x7f605419b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:46:46.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:46.383+0000 7f6059f7a700 1 -- 172.21.15.158:0/3506180596 >> 172.21.15.158:0/3506180596 conn(0x7f60540fa420 msgr2=0x7f60540fc6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:46:46.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:46.383+0000 7f6059f7a700 1 -- 172.21.15.158:0/3506180596 shutdown_connections 2023-07-20T02:46:46.394 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:46:46.383+0000 7f6059f7a700 1 -- 172.21.15.158:0/3506180596 wait complete. 2023-07-20T02:46:46.767 DEBUG:teuthology.orchestra.run.smithi158:osd.1> sudo journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.1.service 2023-07-20T02:46:46.770 INFO:tasks.cephadm:Deploying osd.2 on smithi188 with /dev/vg_nvme/lv_4... 2023-07-20T02:46:46.770 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-20T02:46:46.801 INFO:journalctl@ceph.osd.1.smithi158.stdout:-- Logs begin at Thu 2023-07-20 02:10:12 UTC. -- 2023-07-20T02:46:46.802 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:46 smithi188 ceph-mon[129553]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:46:46.802 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:46 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:46.802 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:46 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:46.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:46 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:46.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:46 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:46:46.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:46 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:46.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:46 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:46.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:46 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:46.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:46 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:46:46.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:46 smithi158 ceph-mon[128925]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:46:46.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:46 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:46.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:46 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:46.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:46 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:46.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:46 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:46:46.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:46 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:46.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:46 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:46.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:46 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:46.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:46 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:46:46.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:46 smithi007 ceph-mon[126353]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:46:46.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:46 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:46.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:46 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:46.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:46 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:46.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:46 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:46:46.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:46 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:46.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:46 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:46.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:46 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:46.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:46 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:46:47.334 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T02:46:47.825 INFO:journalctl@ceph.osd.1.smithi158.stdout:Jul 20 02:46:47 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-1[134008]: 2023-07-20T02:46:47.728+0000 7fbbe2e15640 -1 osd.1 0 log_to_monitors true 2023-07-20T02:46:47.901 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:47 smithi188 ceph-mon[129553]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:46:47.901 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:47 smithi188 ceph-mon[129553]: from='osd.1 [v2:172.21.15.158:6800/2796771612,v1:172.21.15.158:6801/2796771612]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T02:46:47.902 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:47 smithi188 ceph-mon[129553]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T02:46:48.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:47 smithi158 ceph-mon[128925]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:46:48.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:47 smithi158 ceph-mon[128925]: from='osd.1 [v2:172.21.15.158:6800/2796771612,v1:172.21.15.158:6801/2796771612]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T02:46:48.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:47 smithi158 ceph-mon[128925]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T02:46:48.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:47 smithi007 ceph-mon[126353]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:46:48.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:47 smithi007 ceph-mon[126353]: from='osd.1 [v2:172.21.15.158:6800/2796771612,v1:172.21.15.158:6801/2796771612]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T02:46:48.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:47 smithi007 ceph-mon[126353]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T02:46:49.070 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-20T02:46:49.071 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-20T02:46:49.071 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2023-07-20T02:46:49.071 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2023-07-20T02:46:49.072 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.014647 s, 716 MB/s 2023-07-20T02:46:49.072 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2023-07-20T02:46:49.120 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:48 smithi188 ceph-mon[129553]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-20T02:46:49.120 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:48 smithi188 ceph-mon[129553]: osdmap e11: 2 total, 1 up, 2 in 2023-07-20T02:46:49.120 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:48 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T02:46:49.121 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:48 smithi188 ceph-mon[129553]: from='osd.1 [v2:172.21.15.158:6800/2796771612,v1:172.21.15.158:6801/2796771612]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]: dispatch 2023-07-20T02:46:49.121 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:48 smithi188 ceph-mon[129553]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]: dispatch 2023-07-20T02:46:49.137 INFO:journalctl@ceph.osd.1.smithi158.stdout:Jul 20 02:46:48 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-1[134008]: 2023-07-20T02:46:48.825+0000 7fbbd848d700 -1 osd.1 0 waiting for initial osdmap 2023-07-20T02:46:49.137 INFO:journalctl@ceph.osd.1.smithi158.stdout:Jul 20 02:46:48 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-1[134008]: 2023-07-20T02:46:48.829+0000 7fbbd4283700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T02:46:49.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:48 smithi158 ceph-mon[128925]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-20T02:46:49.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:48 smithi158 ceph-mon[128925]: osdmap e11: 2 total, 1 up, 2 in 2023-07-20T02:46:49.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:48 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T02:46:49.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:48 smithi158 ceph-mon[128925]: from='osd.1 [v2:172.21.15.158:6800/2796771612,v1:172.21.15.158:6801/2796771612]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]: dispatch 2023-07-20T02:46:49.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:48 smithi158 ceph-mon[128925]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]: dispatch 2023-07-20T02:46:49.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:48 smithi007 ceph-mon[126353]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-20T02:46:49.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:48 smithi007 ceph-mon[126353]: osdmap e11: 2 total, 1 up, 2 in 2023-07-20T02:46:49.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:48 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T02:46:49.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:48 smithi007 ceph-mon[126353]: from='osd.1 [v2:172.21.15.158:6800/2796771612,v1:172.21.15.158:6801/2796771612]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]: dispatch 2023-07-20T02:46:49.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:48 smithi007 ceph-mon[126353]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]: dispatch 2023-07-20T02:46:49.579 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi188:vg_nvme/lv_4 2023-07-20T02:46:50.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:49 smithi158 ceph-mon[128925]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]': finished 2023-07-20T02:46:50.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:49 smithi158 ceph-mon[128925]: osdmap e12: 2 total, 1 up, 2 in 2023-07-20T02:46:50.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:49 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T02:46:50.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:49 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T02:46:50.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:49 smithi158 ceph-mon[128925]: Detected new or changed devices on smithi158 2023-07-20T02:46:50.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:49 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:50.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:49 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:50.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:49 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:46:50.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:49 smithi158 ceph-mon[128925]: Adjusting osd_memory_target on smithi158 to 17168M 2023-07-20T02:46:50.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:49 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:50.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:49 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:50.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:49 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:46:50.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:49 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:50.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:49 smithi158 ceph-mon[128925]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:46:50.170 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T02:46:50.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:49 smithi007 ceph-mon[126353]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]': finished 2023-07-20T02:46:50.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:49 smithi007 ceph-mon[126353]: osdmap e12: 2 total, 1 up, 2 in 2023-07-20T02:46:50.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:49 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T02:46:50.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:49 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T02:46:50.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:49 smithi007 ceph-mon[126353]: Detected new or changed devices on smithi158 2023-07-20T02:46:50.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:49 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:50.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:49 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:50.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:49 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:46:50.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:49 smithi007 ceph-mon[126353]: Adjusting osd_memory_target on smithi158 to 17168M 2023-07-20T02:46:50.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:49 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:50.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:49 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:50.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:49 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:46:50.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:49 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:50.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:49 smithi007 ceph-mon[126353]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:46:50.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:49 smithi188 ceph-mon[129553]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]': finished 2023-07-20T02:46:50.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:49 smithi188 ceph-mon[129553]: osdmap e12: 2 total, 1 up, 2 in 2023-07-20T02:46:50.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:49 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T02:46:50.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:49 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T02:46:50.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:49 smithi188 ceph-mon[129553]: Detected new or changed devices on smithi158 2023-07-20T02:46:50.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:49 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:50.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:49 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:50.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:49 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:46:50.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:49 smithi188 ceph-mon[129553]: Adjusting osd_memory_target on smithi158 to 17168M 2023-07-20T02:46:50.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:49 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:50.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:49 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:50.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:49 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:46:50.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:49 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:46:50.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:49 smithi188 ceph-mon[129553]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T02:46:50.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:50 smithi188 ceph-mon[129553]: purged_snaps scrub starts 2023-07-20T02:46:50.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:50 smithi188 ceph-mon[129553]: purged_snaps scrub ok 2023-07-20T02:46:50.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:50 smithi188 ceph-mon[129553]: osd.1 [v2:172.21.15.158:6800/2796771612,v1:172.21.15.158:6801/2796771612] boot 2023-07-20T02:46:50.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:50 smithi188 ceph-mon[129553]: osdmap e13: 2 total, 2 up, 2 in 2023-07-20T02:46:50.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:50 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T02:46:51.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:50 smithi158 ceph-mon[128925]: purged_snaps scrub starts 2023-07-20T02:46:51.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:50 smithi158 ceph-mon[128925]: purged_snaps scrub ok 2023-07-20T02:46:51.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:50 smithi158 ceph-mon[128925]: osd.1 [v2:172.21.15.158:6800/2796771612,v1:172.21.15.158:6801/2796771612] boot 2023-07-20T02:46:51.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:50 smithi158 ceph-mon[128925]: osdmap e13: 2 total, 2 up, 2 in 2023-07-20T02:46:51.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:50 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T02:46:51.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:50 smithi007 ceph-mon[126353]: purged_snaps scrub starts 2023-07-20T02:46:51.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:50 smithi007 ceph-mon[126353]: purged_snaps scrub ok 2023-07-20T02:46:51.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:50 smithi007 ceph-mon[126353]: osd.1 [v2:172.21.15.158:6800/2796771612,v1:172.21.15.158:6801/2796771612] boot 2023-07-20T02:46:51.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:50 smithi007 ceph-mon[126353]: osdmap e13: 2 total, 2 up, 2 in 2023-07-20T02:46:51.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:50 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T02:46:51.425 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.413+0000 7f7a18381700 1 -- 172.21.15.188:0/3980784881 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a10100ce0 msgr2=0x7f7a10101100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:46:51.425 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.413+0000 7f7a18381700 1 --2- 172.21.15.188:0/3980784881 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a10100ce0 0x7f7a10101100 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7a00009990 tx=0x7f7a00009ca0 comp rx=0 tx=0).stop 2023-07-20T02:46:51.425 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.414+0000 7f7a18381700 1 -- 172.21.15.188:0/3980784881 shutdown_connections 2023-07-20T02:46:51.425 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.414+0000 7f7a18381700 1 --2- 172.21.15.188:0/3980784881 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a10100330 0x7f7a10107bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:46:51.425 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.414+0000 7f7a18381700 1 --2- 172.21.15.188:0/3980784881 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a10100ce0 0x7f7a10101100 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:46:51.426 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.414+0000 7f7a18381700 1 --2- 172.21.15.188:0/3980784881 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a100ff990 0x7f7a100ffdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:46:51.426 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.414+0000 7f7a18381700 1 -- 172.21.15.188:0/3980784881 >> 172.21.15.188:0/3980784881 conn(0x7f7a100fa420 msgr2=0x7f7a100fc860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:46:51.426 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.414+0000 7f7a18381700 1 -- 172.21.15.188:0/3980784881 shutdown_connections 2023-07-20T02:46:51.426 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.414+0000 7f7a18381700 1 -- 172.21.15.188:0/3980784881 wait complete. 2023-07-20T02:46:51.426 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.414+0000 7f7a18381700 1 Processor -- start 2023-07-20T02:46:51.426 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.414+0000 7f7a18381700 1 -- start start 2023-07-20T02:46:51.426 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.414+0000 7f7a18381700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a100ff990 0x7f7a10195490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:46:51.427 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.414+0000 7f7a18381700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a10100330 0x7f7a101959d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:46:51.427 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.414+0000 7f7a18381700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a10100ce0 0x7f7a1019ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:46:51.427 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.414+0000 7f7a18381700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1019d020 con 0x7f7a10100ce0 2023-07-20T02:46:51.427 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.414+0000 7f7a18381700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1019d160 con 0x7f7a10100330 2023-07-20T02:46:51.427 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.414+0000 7f7a18381700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1019d2f0 con 0x7f7a100ff990 2023-07-20T02:46:51.427 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.415+0000 7f7a1611d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a100ff990 0x7f7a10195490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:46:51.427 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.415+0000 7f7a1611d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a100ff990 0x7f7a10195490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:34850/0 (socket says 172.21.15.188:34850) 2023-07-20T02:46:51.428 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.415+0000 7f7a1611d700 1 -- 172.21.15.188:0/3264954474 learned_addr learned my addr 172.21.15.188:0/3264954474 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-07-20T02:46:51.428 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.415+0000 7f7a1591c700 1 --2- 172.21.15.188:0/3264954474 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a10100330 0x7f7a101959d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:46:51.428 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.415+0000 7f7a1691e700 1 --2- 172.21.15.188:0/3264954474 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a10100ce0 0x7f7a1019ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:46:51.428 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.415+0000 7f7a1611d700 1 -- 172.21.15.188:0/3264954474 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a10100330 msgr2=0x7f7a101959d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:46:51.428 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.415+0000 7f7a1611d700 1 --2- 172.21.15.188:0/3264954474 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a10100330 0x7f7a101959d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:46:51.428 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.415+0000 7f7a1611d700 1 -- 172.21.15.188:0/3264954474 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a10100ce0 msgr2=0x7f7a1019ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:46:51.428 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.415+0000 7f7a1611d700 1 --2- 172.21.15.188:0/3264954474 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a10100ce0 0x7f7a1019ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:46:51.428 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.415+0000 7f7a1611d700 1 -- 172.21.15.188:0/3264954474 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a000095f0 con 0x7f7a100ff990 2023-07-20T02:46:51.429 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.415+0000 7f7a1691e700 1 --2- 172.21.15.188:0/3264954474 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a10100ce0 0x7f7a1019ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T02:46:51.429 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.415+0000 7f7a1611d700 1 --2- 172.21.15.188:0/3264954474 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a100ff990 0x7f7a10195490 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7a0000f690 tx=0x7f7a0000f6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:46:51.429 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.415+0000 7f7a077fe700 1 -- 172.21.15.188:0/3264954474 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0001d070 con 0x7f7a100ff990 2023-07-20T02:46:51.429 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.415+0000 7f7a077fe700 1 -- 172.21.15.188:0/3264954474 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7a00004cf0 con 0x7f7a100ff990 2023-07-20T02:46:51.429 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.415+0000 7f7a077fe700 1 -- 172.21.15.188:0/3264954474 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a00005480 con 0x7f7a100ff990 2023-07-20T02:46:51.429 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.415+0000 7f7a18381700 1 -- 172.21.15.188:0/3264954474 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a1019d570 con 0x7f7a100ff990 2023-07-20T02:46:51.429 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.415+0000 7f7a18381700 1 -- 172.21.15.188:0/3264954474 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a1019d8d0 con 0x7f7a100ff990 2023-07-20T02:46:51.430 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.416+0000 7f7a18381700 1 -- 172.21.15.188:0/3264954474 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a10106de0 con 0x7f7a100ff990 2023-07-20T02:46:51.430 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.417+0000 7f7a077fe700 1 -- 172.21.15.188:0/3264954474 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 15) v1 ==== 135658+0+0 (secure 0 0 0) 0x7f7a0000fba0 con 0x7f7a100ff990 2023-07-20T02:46:51.431 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.418+0000 7f7a077fe700 1 --2- 172.21.15.188:0/3264954474 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f79fc0a10c0 0x7f79fc0a3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:46:51.431 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.418+0000 7f7a077fe700 1 -- 172.21.15.188:0/3264954474 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7f7a000cc3a0 con 0x7f7a100ff990 2023-07-20T02:46:51.431 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.418+0000 7f7a1591c700 1 --2- 172.21.15.188:0/3264954474 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f79fc0a10c0 0x7f79fc0a3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:46:51.431 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.419+0000 7f7a1591c700 1 --2- 172.21.15.188:0/3264954474 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f79fc0a10c0 0x7f79fc0a3580 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c006fd0 tx=0x7f7a0c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:46:51.432 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.422+0000 7f7a077fe700 1 -- 172.21.15.188:0/3264954474 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7a0005b780 con 0x7f7a100ff990 2023-07-20T02:46:51.625 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:46:51.615+0000 7f7a18381700 1 -- 172.21.15.188:0/3264954474 --> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a10060bb0 con 0x7f79fc0a10c0 2023-07-20T02:46:52.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:51 smithi007 ceph-mon[126353]: osdmap e14: 2 total, 2 up, 2 in 2023-07-20T02:46:52.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:51 smithi007 ceph-mon[126353]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:46:52.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:51 smithi007 ceph-mon[126353]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:46:52.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:51 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:46:52.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:51 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:46:52.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:51 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:52.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:51 smithi188 ceph-mon[129553]: osdmap e14: 2 total, 2 up, 2 in 2023-07-20T02:46:52.210 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:51 smithi188 ceph-mon[129553]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:46:52.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:51 smithi188 ceph-mon[129553]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:46:52.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:51 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:46:52.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:51 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:46:52.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:51 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:52.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:51 smithi158 ceph-mon[128925]: osdmap e14: 2 total, 2 up, 2 in 2023-07-20T02:46:52.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:51 smithi158 ceph-mon[128925]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:46:52.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:51 smithi158 ceph-mon[128925]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:46:52.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:51 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:46:52.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:51 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:46:52.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:51 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:46:54.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:54 smithi158 ceph-mon[128925]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:46:54.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:54 smithi158 ceph-mon[128925]: from='client.? 172.21.15.188:0/2482305161' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c777fe6-d003-49d0-af37-f63fca37ff06"}]: dispatch 2023-07-20T02:46:54.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:54 smithi158 ceph-mon[128925]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c777fe6-d003-49d0-af37-f63fca37ff06"}]: dispatch 2023-07-20T02:46:54.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:54 smithi158 ceph-mon[128925]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c777fe6-d003-49d0-af37-f63fca37ff06"}]': finished 2023-07-20T02:46:54.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:54 smithi158 ceph-mon[128925]: osdmap e15: 3 total, 2 up, 3 in 2023-07-20T02:46:54.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:54 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T02:46:54.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:54 smithi007 ceph-mon[126353]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:46:54.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:54 smithi007 ceph-mon[126353]: from='client.? 172.21.15.188:0/2482305161' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c777fe6-d003-49d0-af37-f63fca37ff06"}]: dispatch 2023-07-20T02:46:54.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:54 smithi007 ceph-mon[126353]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c777fe6-d003-49d0-af37-f63fca37ff06"}]: dispatch 2023-07-20T02:46:54.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:54 smithi007 ceph-mon[126353]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c777fe6-d003-49d0-af37-f63fca37ff06"}]': finished 2023-07-20T02:46:54.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:54 smithi007 ceph-mon[126353]: osdmap e15: 3 total, 2 up, 3 in 2023-07-20T02:46:54.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:54 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T02:46:54.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:54 smithi188 ceph-mon[129553]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:46:54.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:54 smithi188 ceph-mon[129553]: from='client.? 172.21.15.188:0/2482305161' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c777fe6-d003-49d0-af37-f63fca37ff06"}]: dispatch 2023-07-20T02:46:54.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:54 smithi188 ceph-mon[129553]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c777fe6-d003-49d0-af37-f63fca37ff06"}]: dispatch 2023-07-20T02:46:54.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:54 smithi188 ceph-mon[129553]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c777fe6-d003-49d0-af37-f63fca37ff06"}]': finished 2023-07-20T02:46:54.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:54 smithi188 ceph-mon[129553]: osdmap e15: 3 total, 2 up, 3 in 2023-07-20T02:46:54.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:54 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T02:46:55.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:55 smithi158 ceph-mon[128925]: from='client.? 172.21.15.188:0/1115932334' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:46:55.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:55 smithi007 ceph-mon[126353]: from='client.? 172.21.15.188:0/1115932334' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:46:55.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:55 smithi188 ceph-mon[129553]: from='client.? 172.21.15.188:0/1115932334' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:46:56.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:56 smithi158 ceph-mon[128925]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:46:56.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:56 smithi007 ceph-mon[126353]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:46:56.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:56 smithi188 ceph-mon[129553]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:46:58.092 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:46:57 smithi188 ceph-mon[129553]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:46:58.137 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:46:57 smithi158 ceph-mon[128925]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:46:58.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:46:57 smithi007 ceph-mon[126353]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:47:00.710 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:00 smithi188 ceph-mon[129553]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:47:00.710 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:00 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-20T02:47:00.710 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:00 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:47:00.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:00 smithi158 ceph-mon[128925]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:47:00.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:00 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-20T02:47:00.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:00 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:47:00.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:00 smithi007 ceph-mon[126353]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:47:00.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:00 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-20T02:47:00.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:00 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:47:01.710 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:01 smithi188 ceph-mon[129553]: Deploying daemon osd.2 on smithi188 2023-07-20T02:47:01.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:01 smithi158 ceph-mon[128925]: Deploying daemon osd.2 on smithi188 2023-07-20T02:47:01.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:01 smithi007 ceph-mon[126353]: Deploying daemon osd.2 on smithi188 2023-07-20T02:47:02.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:02 smithi158 ceph-mon[128925]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:47:02.902 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:02 smithi188 ceph-mon[129553]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:47:02.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:02 smithi007 ceph-mon[126353]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:47:04.858 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:04 smithi188 ceph-mon[129553]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:47:04.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:04 smithi158 ceph-mon[128925]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:47:04.944 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:04 smithi007 ceph-mon[126353]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:47:06.075 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:05 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:06.075 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:05 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:06.075 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:05 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:47:06.075 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:05 smithi188 ceph-mon[129553]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:47:06.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:05 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:06.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:05 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:06.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:05 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:47:06.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:05 smithi007 ceph-mon[126353]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:47:06.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:05 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:06.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:05 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:06.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:05 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:47:06.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:05 smithi158 ceph-mon[128925]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:47:08.094 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 2 on host 'smithi188' 2023-07-20T02:47:08.094 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:47:08.084+0000 7f7a077fe700 1 -- 172.21.15.188:0/3264954474 <== mgr.14152 v2:172.21.15.7:6800/1354542137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7a10060bb0 con 0x7f79fc0a10c0 2023-07-20T02:47:08.096 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:47:08.087+0000 7f7a18381700 1 -- 172.21.15.188:0/3264954474 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f79fc0a10c0 msgr2=0x7f79fc0a3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:47:08.096 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:47:08.087+0000 7f7a18381700 1 --2- 172.21.15.188:0/3264954474 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f79fc0a10c0 0x7f79fc0a3580 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c006fd0 tx=0x7f7a0c008040 comp rx=0 tx=0).stop 2023-07-20T02:47:08.096 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:47:08.087+0000 7f7a18381700 1 -- 172.21.15.188:0/3264954474 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a100ff990 msgr2=0x7f7a10195490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:47:08.097 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:47:08.087+0000 7f7a18381700 1 --2- 172.21.15.188:0/3264954474 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a100ff990 0x7f7a10195490 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7a0000f690 tx=0x7f7a0000f6c0 comp rx=0 tx=0).stop 2023-07-20T02:47:08.097 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:47:08.087+0000 7f7a18381700 1 -- 172.21.15.188:0/3264954474 shutdown_connections 2023-07-20T02:47:08.097 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:47:08.087+0000 7f7a18381700 1 --2- 172.21.15.188:0/3264954474 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f79fc0a10c0 0x7f79fc0a3580 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:47:08.097 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:47:08.087+0000 7f7a18381700 1 --2- 172.21.15.188:0/3264954474 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a10100ce0 0x7f7a1019ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:47:08.097 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:47:08.087+0000 7f7a18381700 1 --2- 172.21.15.188:0/3264954474 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a100ff990 0x7f7a10195490 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:47:08.097 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:47:08.087+0000 7f7a18381700 1 --2- 172.21.15.188:0/3264954474 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7a10100330 0x7f7a101959d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:47:08.097 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:47:08.087+0000 7f7a18381700 1 -- 172.21.15.188:0/3264954474 >> 172.21.15.188:0/3264954474 conn(0x7f7a100fa420 msgr2=0x7f7a10103f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:47:08.098 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:47:08.087+0000 7f7a18381700 1 -- 172.21.15.188:0/3264954474 shutdown_connections 2023-07-20T02:47:08.098 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:47:08.087+0000 7f7a18381700 1 -- 172.21.15.188:0/3264954474 wait complete. 2023-07-20T02:47:08.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:08 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:08.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:08 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:08.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:08 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:47:08.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:08 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:47:08.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:08 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:08.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:08 smithi158 ceph-mon[128925]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:47:08.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:08 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:08.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:08 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:08.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:08 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:47:08.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:08 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:47:08.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:08 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:08.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:08 smithi007 ceph-mon[126353]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:47:08.460 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:08 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:08.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:08 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:08.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:08 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:47:08.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:08 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:47:08.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:08 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:08.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:08 smithi188 ceph-mon[129553]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:47:08.551 DEBUG:teuthology.orchestra.run.smithi188:osd.2> sudo journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.2.service 2023-07-20T02:47:08.554 INFO:tasks.cephadm:Deploying osd.3 on smithi007 with /dev/vg_nvme/lv_3... 2023-07-20T02:47:08.555 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-20T02:47:08.584 INFO:journalctl@ceph.osd.2.smithi188.stdout:-- Logs begin at Thu 2023-07-20 02:09:43 UTC. -- 2023-07-20T02:47:08.960 INFO:journalctl@ceph.osd.2.smithi188.stdout:Jul 20 02:47:08 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-2[135329]: 2023-07-20T02:47:08.766+0000 7fc13cac6640 -1 osd.2 0 log_to_monitors true 2023-07-20T02:47:09.295 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:47:09.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:09 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:09.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:09 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:09.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:09 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:47:09.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:09 smithi158 ceph-mon[128925]: from='osd.2 [v2:172.21.15.188:6800/2928307258,v1:172.21.15.188:6801/2928307258]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T02:47:09.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:09 smithi158 ceph-mon[128925]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T02:47:09.405 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:09 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:09.405 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:09 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:09.405 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:09 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:47:09.406 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:09 smithi188 ceph-mon[129553]: from='osd.2 [v2:172.21.15.188:6800/2928307258,v1:172.21.15.188:6801/2928307258]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T02:47:09.406 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:09 smithi188 ceph-mon[129553]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T02:47:09.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:09 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:09.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:09 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:09.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:09 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:47:09.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:09 smithi007 ceph-mon[126353]: from='osd.2 [v2:172.21.15.188:6800/2928307258,v1:172.21.15.188:6801/2928307258]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T02:47:09.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:09 smithi007 ceph-mon[126353]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T02:47:10.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:10 smithi007 ceph-mon[126353]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-20T02:47:10.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:10 smithi007 ceph-mon[126353]: osdmap e16: 3 total, 2 up, 3 in 2023-07-20T02:47:10.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:10 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T02:47:10.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:10 smithi007 ceph-mon[126353]: from='osd.2 [v2:172.21.15.188:6800/2928307258,v1:172.21.15.188:6801/2928307258]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-07-20T02:47:10.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:10 smithi007 ceph-mon[126353]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-07-20T02:47:10.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:10 smithi007 ceph-mon[126353]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:47:10.210 INFO:journalctl@ceph.osd.2.smithi188.stdout:Jul 20 02:47:10 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-2[135329]: 2023-07-20T02:47:10.091+0000 7fc13213e700 -1 osd.2 0 waiting for initial osdmap 2023-07-20T02:47:10.211 INFO:journalctl@ceph.osd.2.smithi188.stdout:Jul 20 02:47:10 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-2[135329]: 2023-07-20T02:47:10.096+0000 7fc12c731700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T02:47:10.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:10 smithi188 ceph-mon[129553]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-20T02:47:10.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:10 smithi188 ceph-mon[129553]: osdmap e16: 3 total, 2 up, 3 in 2023-07-20T02:47:10.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:10 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T02:47:10.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:10 smithi188 ceph-mon[129553]: from='osd.2 [v2:172.21.15.188:6800/2928307258,v1:172.21.15.188:6801/2928307258]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-07-20T02:47:10.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:10 smithi188 ceph-mon[129553]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-07-20T02:47:10.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:10 smithi188 ceph-mon[129553]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:47:10.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:10 smithi158 ceph-mon[128925]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-20T02:47:10.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:10 smithi158 ceph-mon[128925]: osdmap e16: 3 total, 2 up, 3 in 2023-07-20T02:47:10.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:10 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T02:47:10.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:10 smithi158 ceph-mon[128925]: from='osd.2 [v2:172.21.15.188:6800/2928307258,v1:172.21.15.188:6801/2928307258]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-07-20T02:47:10.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:10 smithi158 ceph-mon[128925]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-07-20T02:47:10.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:10 smithi158 ceph-mon[128925]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:47:11.384 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-20T02:47:11.385 INFO:teuthology.orchestra.run.smithi007.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-20T02:47:11.385 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10+0 records in 2023-07-20T02:47:11.385 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2023-07-20T02:47:11.385 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0300104 s, 349 MB/s 2023-07-20T02:47:11.385 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping successful for: 2023-07-20T02:47:11.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:11 smithi158 ceph-mon[128925]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2023-07-20T02:47:11.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:11 smithi158 ceph-mon[128925]: osdmap e17: 3 total, 2 up, 3 in 2023-07-20T02:47:11.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:11 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T02:47:11.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:11 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T02:47:11.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:11 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:11.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:11 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:11.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:11 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:47:11.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:11 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:11.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:11 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:47:11.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:11 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:47:11.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:11 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:11.421 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:11 smithi007 ceph-mon[126353]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2023-07-20T02:47:11.421 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:11 smithi007 ceph-mon[126353]: osdmap e17: 3 total, 2 up, 3 in 2023-07-20T02:47:11.421 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:11 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T02:47:11.422 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:11 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T02:47:11.422 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:11 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:11.422 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:11 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:11.422 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:11 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:47:11.422 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:11 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:11.422 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:11 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:47:11.422 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:11 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:47:11.422 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:11 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:11.460 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:11 smithi188 ceph-mon[129553]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2023-07-20T02:47:11.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:11 smithi188 ceph-mon[129553]: osdmap e17: 3 total, 2 up, 3 in 2023-07-20T02:47:11.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:11 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T02:47:11.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:11 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T02:47:11.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:11 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:11.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:11 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:11.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:11 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:47:11.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:11 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:11.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:11 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:47:11.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:11 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:47:11.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:11 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:11.875 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi007:vg_nvme/lv_3 2023-07-20T02:47:12.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:12 smithi158 ceph-mon[128925]: purged_snaps scrub starts 2023-07-20T02:47:12.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:12 smithi158 ceph-mon[128925]: purged_snaps scrub ok 2023-07-20T02:47:12.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:12 smithi158 ceph-mon[128925]: Detected new or changed devices on smithi188 2023-07-20T02:47:12.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:12 smithi158 ceph-mon[128925]: Adjusting osd_memory_target on smithi188 to 17168M 2023-07-20T02:47:12.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:12 smithi158 ceph-mon[128925]: osd.2 [v2:172.21.15.188:6800/2928307258,v1:172.21.15.188:6801/2928307258] boot 2023-07-20T02:47:12.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:12 smithi158 ceph-mon[128925]: osdmap e18: 3 total, 3 up, 3 in 2023-07-20T02:47:12.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:12 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T02:47:12.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:12 smithi158 ceph-mon[128925]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:47:12.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:12 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-20T02:47:12.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:12 smithi007 ceph-mon[126353]: purged_snaps scrub starts 2023-07-20T02:47:12.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:12 smithi007 ceph-mon[126353]: purged_snaps scrub ok 2023-07-20T02:47:12.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:12 smithi007 ceph-mon[126353]: Detected new or changed devices on smithi188 2023-07-20T02:47:12.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:12 smithi007 ceph-mon[126353]: Adjusting osd_memory_target on smithi188 to 17168M 2023-07-20T02:47:12.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:12 smithi007 ceph-mon[126353]: osd.2 [v2:172.21.15.188:6800/2928307258,v1:172.21.15.188:6801/2928307258] boot 2023-07-20T02:47:12.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:12 smithi007 ceph-mon[126353]: osdmap e18: 3 total, 3 up, 3 in 2023-07-20T02:47:12.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:12 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T02:47:12.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:12 smithi007 ceph-mon[126353]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:47:12.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:12 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-20T02:47:12.460 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:12 smithi188 ceph-mon[129553]: purged_snaps scrub starts 2023-07-20T02:47:12.460 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:12 smithi188 ceph-mon[129553]: purged_snaps scrub ok 2023-07-20T02:47:12.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:12 smithi188 ceph-mon[129553]: Detected new or changed devices on smithi188 2023-07-20T02:47:12.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:12 smithi188 ceph-mon[129553]: Adjusting osd_memory_target on smithi188 to 17168M 2023-07-20T02:47:12.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:12 smithi188 ceph-mon[129553]: osd.2 [v2:172.21.15.188:6800/2928307258,v1:172.21.15.188:6801/2928307258] boot 2023-07-20T02:47:12.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:12 smithi188 ceph-mon[129553]: osdmap e18: 3 total, 3 up, 3 in 2023-07-20T02:47:12.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:12 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T02:47:12.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:12 smithi188 ceph-mon[129553]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T02:47:12.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:12 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-20T02:47:12.604 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:47:13.176 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:13 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-20T02:47:13.176 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:13 smithi007 ceph-mon[126353]: osdmap e19: 3 total, 3 up, 3 in 2023-07-20T02:47:13.176 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:13 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-20T02:47:13.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:13 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-20T02:47:13.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:13 smithi158 ceph-mon[128925]: osdmap e19: 3 total, 3 up, 3 in 2023-07-20T02:47:13.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:13 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-20T02:47:13.460 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:13 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-20T02:47:13.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:13 smithi188 ceph-mon[129553]: osdmap e19: 3 total, 3 up, 3 in 2023-07-20T02:47:13.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:13 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-20T02:47:14.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.165+0000 7f5f10c00700 1 -- 172.21.15.7:0/4217368003 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f0c100900 msgr2=0x7f5f0c100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:47:14.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.165+0000 7f5f10c00700 1 --2- 172.21.15.7:0/4217368003 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f0c100900 0x7f5f0c100d60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5f000099c0 tx=0x7f5f00009cd0 comp rx=0 tx=0).stop 2023-07-20T02:47:14.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.166+0000 7f5f10c00700 1 -- 172.21.15.7:0/4217368003 shutdown_connections 2023-07-20T02:47:14.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.166+0000 7f5f10c00700 1 --2- 172.21.15.7:0/4217368003 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f0c1012a0 0x7f5f0c107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:47:14.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.166+0000 7f5f10c00700 1 --2- 172.21.15.7:0/4217368003 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f0c0ff700 0x7f5f0c0ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:47:14.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.166+0000 7f5f10c00700 1 --2- 172.21.15.7:0/4217368003 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f0c100900 0x7f5f0c100d60 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:47:14.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.166+0000 7f5f10c00700 1 -- 172.21.15.7:0/4217368003 >> 172.21.15.7:0/4217368003 conn(0x7f5f0c0fac80 msgr2=0x7f5f0c0fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:47:14.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.166+0000 7f5f10c00700 1 -- 172.21.15.7:0/4217368003 shutdown_connections 2023-07-20T02:47:14.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.166+0000 7f5f10c00700 1 -- 172.21.15.7:0/4217368003 wait complete. 2023-07-20T02:47:14.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.167+0000 7f5f10c00700 1 Processor -- start 2023-07-20T02:47:14.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.167+0000 7f5f10c00700 1 -- start start 2023-07-20T02:47:14.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.168+0000 7f5f10c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f0c0ff700 0x7f5f0c10a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:47:14.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.168+0000 7f5f10c00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f0c100900 0x7f5f0c108660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:47:14.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.168+0000 7f5f10c00700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f0c1012a0 0x7f5f0c108ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:47:14.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.168+0000 7f5f10c00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0c109320 con 0x7f5f0c100900 2023-07-20T02:47:14.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.168+0000 7f5f10c00700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0c109490 con 0x7f5f0c1012a0 2023-07-20T02:47:14.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.168+0000 7f5f10c00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0c1a0520 con 0x7f5f0c0ff700 2023-07-20T02:47:14.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.168+0000 7f5f09d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f0c100900 0x7f5f0c108660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:47:14.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.168+0000 7f5f09d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f0c100900 0x7f5f0c108660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37042/0 (socket says 172.21.15.7:37042) 2023-07-20T02:47:14.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.168+0000 7f5f09d9b700 1 -- 172.21.15.7:0/1405689095 learned_addr learned my addr 172.21.15.7:0/1405689095 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:47:14.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.168+0000 7f5f0a59c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f0c0ff700 0x7f5f0c10a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:47:14.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.168+0000 7f5f0ad9d700 1 --2- 172.21.15.7:0/1405689095 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f0c1012a0 0x7f5f0c108ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:47:14.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.169+0000 7f5f0a59c700 1 -- 172.21.15.7:0/1405689095 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f0c1012a0 msgr2=0x7f5f0c108ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:47:14.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.169+0000 7f5f0a59c700 1 --2- 172.21.15.7:0/1405689095 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f0c1012a0 0x7f5f0c108ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:47:14.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.169+0000 7f5f0a59c700 1 -- 172.21.15.7:0/1405689095 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f0c100900 msgr2=0x7f5f0c108660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:47:14.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.169+0000 7f5f0a59c700 1 --2- 172.21.15.7:0/1405689095 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f0c100900 0x7f5f0c108660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:47:14.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.169+0000 7f5f0a59c700 1 -- 172.21.15.7:0/1405689095 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f000095f0 con 0x7f5f0c0ff700 2023-07-20T02:47:14.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.169+0000 7f5f09d9b700 1 --2- 172.21.15.7:0/1405689095 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f0c100900 0x7f5f0c108660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T02:47:14.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.169+0000 7f5f0a59c700 1 --2- 172.21.15.7:0/1405689095 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f0c0ff700 0x7f5f0c10a010 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5efc00e9c0 tx=0x7f5efc00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:47:14.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.169+0000 7f5efb7fe700 1 -- 172.21.15.7:0/1405689095 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5efc004220 con 0x7f5f0c0ff700 2023-07-20T02:47:14.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.169+0000 7f5efb7fe700 1 -- 172.21.15.7:0/1405689095 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5efc004380 con 0x7f5f0c0ff700 2023-07-20T02:47:14.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.169+0000 7f5f10c00700 1 -- 172.21.15.7:0/1405689095 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f0c1a0800 con 0x7f5f0c0ff700 2023-07-20T02:47:14.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.169+0000 7f5efb7fe700 1 -- 172.21.15.7:0/1405689095 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5efc003be0 con 0x7f5f0c0ff700 2023-07-20T02:47:14.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.169+0000 7f5f10c00700 1 -- 172.21.15.7:0/1405689095 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f0c1a0d00 con 0x7f5f0c0ff700 2023-07-20T02:47:14.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.170+0000 7f5f10c00700 1 -- 172.21.15.7:0/1405689095 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f0c10c0d0 con 0x7f5f0c0ff700 2023-07-20T02:47:14.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.173+0000 7f5efb7fe700 1 -- 172.21.15.7:0/1405689095 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 15) v1 ==== 135658+0+0 (secure 0 0 0) 0x7f5efc003d40 con 0x7f5f0c0ff700 2023-07-20T02:47:14.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.174+0000 7f5efb7fe700 1 --2- 172.21.15.7:0/1405689095 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f5ef40a10c0 0x7f5ef40a3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:47:14.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.174+0000 7f5efb7fe700 1 -- 172.21.15.7:0/1405689095 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3237+0+0 (secure 0 0 0) 0x7f5efc014070 con 0x7f5f0c0ff700 2023-07-20T02:47:14.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.174+0000 7f5f09d9b700 1 --2- 172.21.15.7:0/1405689095 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f5ef40a10c0 0x7f5ef40a3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:47:14.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.175+0000 7f5f09d9b700 1 --2- 172.21.15.7:0/1405689095 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f5ef40a10c0 0x7f5ef40a3580 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5f00000c00 tx=0x7f5f00005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:47:14.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.176+0000 7f5efb7fe700 1 -- 172.21.15.7:0/1405689095 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5efc096810 con 0x7f5f0c0ff700 2023-07-20T02:47:14.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:14 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-20T02:47:14.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:14 smithi007 ceph-mon[126353]: osdmap e20: 3 total, 3 up, 3 in 2023-07-20T02:47:14.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:14 smithi007 ceph-mon[126353]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:47:14.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:14 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-20T02:47:14.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:14 smithi158 ceph-mon[128925]: osdmap e20: 3 total, 3 up, 3 in 2023-07-20T02:47:14.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:14 smithi158 ceph-mon[128925]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:47:14.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:14.391+0000 7f5f10c00700 1 -- 172.21.15.7:0/1405689095 --> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f0c1a13a0 con 0x7f5ef40a10c0 2023-07-20T02:47:14.460 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:14 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-20T02:47:14.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:14 smithi188 ceph-mon[129553]: osdmap e20: 3 total, 3 up, 3 in 2023-07-20T02:47:14.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:14 smithi188 ceph-mon[129553]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:47:14.695 INFO:journalctl@ceph.osd.0.smithi007.stdout:Jul 20 02:47:14 smithi007 sudo[142453]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T02:47:14.695 INFO:journalctl@ceph.osd.0.smithi007.stdout:Jul 20 02:47:14 smithi007 sudo[142453]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T02:47:14.695 INFO:journalctl@ceph.osd.0.smithi007.stdout:Jul 20 02:47:14 smithi007 sudo[142453]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T02:47:14.695 INFO:journalctl@ceph.osd.0.smithi007.stdout:Jul 20 02:47:14 smithi007 sudo[142453]: pam_unix(sudo:session): session closed for user root 2023-07-20T02:47:15.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:15 smithi007 ceph-mon[126353]: osdmap e21: 3 total, 3 up, 3 in 2023-07-20T02:47:15.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:15 smithi007 ceph-mon[126353]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:47:15.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:15 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:47:15.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:15 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:47:15.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:15 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:47:15.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:15 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:15.196 INFO:journalctl@ceph.osd.0.smithi007.stdout:Jul 20 02:47:14 smithi007 sudo[142535]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T02:47:15.196 INFO:journalctl@ceph.osd.0.smithi007.stdout:Jul 20 02:47:14 smithi007 sudo[142535]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T02:47:15.196 INFO:journalctl@ceph.osd.0.smithi007.stdout:Jul 20 02:47:14 smithi007 sudo[142535]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T02:47:15.196 INFO:journalctl@ceph.osd.0.smithi007.stdout:Jul 20 02:47:14 smithi007 sudo[142535]: pam_unix(sudo:session): session closed for user root 2023-07-20T02:47:15.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:15 smithi158 ceph-mon[128925]: osdmap e21: 3 total, 3 up, 3 in 2023-07-20T02:47:15.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:15 smithi158 ceph-mon[128925]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:47:15.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:15 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:47:15.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:15 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:47:15.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:15 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:47:15.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:15 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:15.389 INFO:journalctl@ceph.osd.1.smithi158.stdout:Jul 20 02:47:15 smithi158 sudo[136297]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T02:47:15.389 INFO:journalctl@ceph.osd.1.smithi158.stdout:Jul 20 02:47:15 smithi158 sudo[136297]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T02:47:15.389 INFO:journalctl@ceph.osd.1.smithi158.stdout:Jul 20 02:47:15 smithi158 sudo[136297]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T02:47:15.389 INFO:journalctl@ceph.osd.1.smithi158.stdout:Jul 20 02:47:15 smithi158 sudo[136297]: pam_unix(sudo:session): session closed for user root 2023-07-20T02:47:15.460 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:15 smithi188 ceph-mon[129553]: osdmap e21: 3 total, 3 up, 3 in 2023-07-20T02:47:15.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:15 smithi188 ceph-mon[129553]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:47:15.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:15 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:47:15.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:15 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:47:15.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:15 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:47:15.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:15 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:15.887 INFO:journalctl@ceph.osd.1.smithi158.stdout:Jul 20 02:47:15 smithi158 sudo[136300]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T02:47:15.887 INFO:journalctl@ceph.osd.1.smithi158.stdout:Jul 20 02:47:15 smithi158 sudo[136300]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T02:47:15.888 INFO:journalctl@ceph.osd.1.smithi158.stdout:Jul 20 02:47:15 smithi158 sudo[136300]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T02:47:15.888 INFO:journalctl@ceph.osd.1.smithi158.stdout:Jul 20 02:47:15 smithi158 sudo[136300]: pam_unix(sudo:session): session closed for user root 2023-07-20T02:47:15.931 INFO:journalctl@ceph.osd.2.smithi188.stdout:Jul 20 02:47:15 smithi188 sudo[137581]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T02:47:15.932 INFO:journalctl@ceph.osd.2.smithi188.stdout:Jul 20 02:47:15 smithi188 sudo[137581]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T02:47:15.932 INFO:journalctl@ceph.osd.2.smithi188.stdout:Jul 20 02:47:15 smithi188 sudo[137581]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T02:47:15.932 INFO:journalctl@ceph.osd.2.smithi188.stdout:Jul 20 02:47:15 smithi188 sudo[137581]: pam_unix(sudo:session): session closed for user root 2023-07-20T02:47:16.210 INFO:journalctl@ceph.osd.2.smithi188.stdout:Jul 20 02:47:15 smithi188 sudo[137584]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T02:47:16.211 INFO:journalctl@ceph.osd.2.smithi188.stdout:Jul 20 02:47:15 smithi188 sudo[137584]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T02:47:16.211 INFO:journalctl@ceph.osd.2.smithi188.stdout:Jul 20 02:47:15 smithi188 sudo[137584]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T02:47:16.211 INFO:journalctl@ceph.osd.2.smithi188.stdout:Jul 20 02:47:15 smithi188 sudo[137584]: pam_unix(sudo:session): session closed for user root 2023-07-20T02:47:16.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:16 smithi007 sudo[142679]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T02:47:16.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:16 smithi007 sudo[142679]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T02:47:16.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:16 smithi007 sudo[142679]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T02:47:16.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:16 smithi007 sudo[142679]: pam_unix(sudo:session): session closed for user root 2023-07-20T02:47:17.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:17.061+0000 7f5efb7fe700 1 -- 172.21.15.7:0/1405689095 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f5efc096be0 con 0x7f5f0c0ff700 2023-07-20T02:47:17.321 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:17 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:17.322 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:17 smithi188 ceph-mon[129553]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:47:17.322 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:17 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:17.322 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:17 smithi188 ceph-mon[129553]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T02:47:17.322 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:17 smithi188 sudo[137587]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T02:47:17.322 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:17 smithi188 sudo[137587]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T02:47:17.322 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:17 smithi188 sudo[137587]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T02:47:17.323 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:17 smithi188 sudo[137587]: pam_unix(sudo:session): session closed for user root 2023-07-20T02:47:17.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:17 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:17.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:17 smithi158 ceph-mon[128925]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:47:17.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:17 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:17.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:17 smithi158 ceph-mon[128925]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T02:47:17.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:17 smithi007 sudo[142701]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T02:47:17.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:17 smithi007 sudo[142701]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T02:47:17.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:17 smithi007 sudo[142701]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T02:47:17.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:17 smithi007 sudo[142701]: pam_unix(sudo:session): session closed for user root 2023-07-20T02:47:17.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:17 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:17.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:17 smithi007 ceph-mon[126353]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:47:17.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:17 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:17.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:17 smithi007 ceph-mon[126353]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T02:47:17.710 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:17 smithi188 sudo[137590]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T02:47:17.711 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:17 smithi188 sudo[137590]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T02:47:17.711 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:17 smithi188 sudo[137590]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T02:47:17.711 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:17 smithi188 sudo[137590]: pam_unix(sudo:session): session closed for user root 2023-07-20T02:47:18.051 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:17 smithi158 sudo[136303]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T02:47:18.051 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:17 smithi158 sudo[136303]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T02:47:18.052 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:17 smithi158 sudo[136303]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T02:47:18.052 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:17 smithi158 sudo[136303]: pam_unix(sudo:session): session closed for user root 2023-07-20T02:47:18.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:18 smithi158 sudo[136306]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T02:47:18.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:18 smithi158 sudo[136306]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T02:47:18.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:18 smithi158 sudo[136306]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T02:47:18.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:18 smithi158 sudo[136306]: pam_unix(sudo:session): session closed for user root 2023-07-20T02:47:18.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:18 smithi158 ceph-mon[128925]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T02:47:18.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:18 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:47:18.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:18 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:47:18.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:18 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:47:18.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:18 smithi158 ceph-mon[128925]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T02:47:18.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:18 smithi158 ceph-mon[128925]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-20T02:47:18.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:18 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:47:18.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:18 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:47:18.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:18 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:47:18.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:18 smithi158 ceph-mon[128925]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:47:18.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:18 smithi158 ceph-mon[128925]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T02:47:18.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:18 smithi158 ceph-mon[128925]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T02:47:18.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:18 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:47:18.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:18 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:47:18.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:18 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:47:18.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:18 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/2803544219' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59e29965-018f-4eb6-adbd-1a9e1df84245"}]: dispatch 2023-07-20T02:47:18.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:18 smithi158 ceph-mon[128925]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59e29965-018f-4eb6-adbd-1a9e1df84245"}]: dispatch 2023-07-20T02:47:18.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:18 smithi158 ceph-mon[128925]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T02:47:18.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:18 smithi158 ceph-mon[128925]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59e29965-018f-4eb6-adbd-1a9e1df84245"}]': finished 2023-07-20T02:47:18.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:18 smithi158 ceph-mon[128925]: osdmap e22: 4 total, 3 up, 4 in 2023-07-20T02:47:18.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:18 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T02:47:18.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:18 smithi007 ceph-mon[126353]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T02:47:18.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:18 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:47:18.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:18 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:47:18.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:18 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:47:18.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:18 smithi007 ceph-mon[126353]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T02:47:18.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:18 smithi007 ceph-mon[126353]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-20T02:47:18.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:18 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:47:18.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:18 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:47:18.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:18 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:47:18.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:18 smithi007 ceph-mon[126353]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:47:18.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:18 smithi007 ceph-mon[126353]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T02:47:18.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:18 smithi007 ceph-mon[126353]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T02:47:18.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:18 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:47:18.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:18 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:47:18.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:18 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:47:18.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:18 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/2803544219' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59e29965-018f-4eb6-adbd-1a9e1df84245"}]: dispatch 2023-07-20T02:47:18.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:18 smithi007 ceph-mon[126353]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59e29965-018f-4eb6-adbd-1a9e1df84245"}]: dispatch 2023-07-20T02:47:18.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:18 smithi007 ceph-mon[126353]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T02:47:18.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:18 smithi007 ceph-mon[126353]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59e29965-018f-4eb6-adbd-1a9e1df84245"}]': finished 2023-07-20T02:47:18.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:18 smithi007 ceph-mon[126353]: osdmap e22: 4 total, 3 up, 4 in 2023-07-20T02:47:18.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:18 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T02:47:18.460 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:18 smithi188 ceph-mon[129553]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T02:47:18.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:18 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:47:18.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:18 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:47:18.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:18 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:47:18.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:18 smithi188 ceph-mon[129553]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T02:47:18.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:18 smithi188 ceph-mon[129553]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-20T02:47:18.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:18 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:47:18.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:18 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:47:18.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:18 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:47:18.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:18 smithi188 ceph-mon[129553]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:47:18.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:18 smithi188 ceph-mon[129553]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T02:47:18.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:18 smithi188 ceph-mon[129553]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T02:47:18.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:18 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T02:47:18.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:18 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T02:47:18.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:18 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T02:47:18.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:18 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/2803544219' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59e29965-018f-4eb6-adbd-1a9e1df84245"}]: dispatch 2023-07-20T02:47:18.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:18 smithi188 ceph-mon[129553]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59e29965-018f-4eb6-adbd-1a9e1df84245"}]: dispatch 2023-07-20T02:47:18.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:18 smithi188 ceph-mon[129553]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T02:47:18.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:18 smithi188 ceph-mon[129553]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59e29965-018f-4eb6-adbd-1a9e1df84245"}]': finished 2023-07-20T02:47:18.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:18 smithi188 ceph-mon[129553]: osdmap e22: 4 total, 3 up, 4 in 2023-07-20T02:47:18.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:18 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T02:47:19.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:19 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/762539323' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:47:19.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:19 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/762539323' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:47:19.460 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:19 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/762539323' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:47:20.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:20 smithi158 ceph-mon[128925]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:47:20.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:20 smithi007 ceph-mon[126353]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:47:20.460 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:20 smithi188 ceph-mon[129553]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:47:22.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:22 smithi158 ceph-mon[128925]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2023-07-20T02:47:22.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:22 smithi007 ceph-mon[126353]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2023-07-20T02:47:22.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:22 smithi188 ceph-mon[129553]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2023-07-20T02:47:24.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:24 smithi158 ceph-mon[128925]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-07-20T02:47:24.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:24 smithi007 ceph-mon[126353]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-07-20T02:47:24.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:24 smithi188 ceph-mon[129553]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-07-20T02:47:25.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:25 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-20T02:47:25.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:25 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:47:25.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:25 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-20T02:47:25.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:25 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:47:25.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:25 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-20T02:47:25.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:25 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:47:26.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:26 smithi007 ceph-mon[126353]: Deploying daemon osd.3 on smithi007 2023-07-20T02:47:26.849 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:26 smithi007 ceph-mon[126353]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2023-07-20T02:47:26.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:26 smithi158 ceph-mon[128925]: Deploying daemon osd.3 on smithi007 2023-07-20T02:47:26.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:26 smithi158 ceph-mon[128925]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2023-07-20T02:47:26.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:26 smithi188 ceph-mon[129553]: Deploying daemon osd.3 on smithi007 2023-07-20T02:47:26.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:26 smithi188 ceph-mon[129553]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2023-07-20T02:47:28.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:28 smithi158 ceph-mon[128925]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2023-07-20T02:47:28.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:28 smithi007 ceph-mon[126353]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2023-07-20T02:47:28.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:28 smithi188 ceph-mon[129553]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2023-07-20T02:47:30.823 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:30 smithi007 ceph-mon[126353]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 69 KiB/s wr, 4 op/s 2023-07-20T02:47:30.887 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:30 smithi158 ceph-mon[128925]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 69 KiB/s wr, 4 op/s 2023-07-20T02:47:30.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:30 smithi188 ceph-mon[129553]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 69 KiB/s wr, 4 op/s 2023-07-20T02:47:32.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:32 smithi158 ceph-mon[128925]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2023-07-20T02:47:32.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:32 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:32.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:32 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:32.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:32 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:47:32.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:32 smithi007 ceph-mon[126353]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2023-07-20T02:47:32.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:32 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:32.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:32 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:32.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:32 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:47:32.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:32 smithi188 ceph-mon[129553]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2023-07-20T02:47:32.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:32 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:32.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:32 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:32.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:32 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:47:34.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:34 smithi007 ceph-mon[126353]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:47:34.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:34 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:34.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:34 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:34.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:34 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:47:34.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:34 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:47:34.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:34 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:34.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:34 smithi158 ceph-mon[128925]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:47:34.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:34 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:34.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:34 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:34.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:34 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:47:34.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:34 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:47:34.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:34 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:34.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:34 smithi188 ceph-mon[129553]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:47:34.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:34 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:34.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:34 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:34.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:34 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:47:34.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:34 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:47:34.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:34 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:35.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:35.657+0000 7f5efb7fe700 1 -- 172.21.15.7:0/1405689095 <== mgr.14152 v2:172.21.15.7:6800/1354542137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5f0c1a13a0 con 0x7f5ef40a10c0 2023-07-20T02:47:35.668 INFO:teuthology.orchestra.run.smithi007.stdout:Created osd(s) 3 on host 'smithi007' 2023-07-20T02:47:35.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:35.659+0000 7f5f10c00700 1 -- 172.21.15.7:0/1405689095 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f5ef40a10c0 msgr2=0x7f5ef40a3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:47:35.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:35.659+0000 7f5f10c00700 1 --2- 172.21.15.7:0/1405689095 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f5ef40a10c0 0x7f5ef40a3580 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5f00000c00 tx=0x7f5f00005fb0 comp rx=0 tx=0).stop 2023-07-20T02:47:35.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:35.659+0000 7f5f10c00700 1 -- 172.21.15.7:0/1405689095 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f0c0ff700 msgr2=0x7f5f0c10a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:47:35.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:35.659+0000 7f5f10c00700 1 --2- 172.21.15.7:0/1405689095 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f0c0ff700 0x7f5f0c10a010 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5efc00e9c0 tx=0x7f5efc00ecd0 comp rx=0 tx=0).stop 2023-07-20T02:47:35.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:35.659+0000 7f5f10c00700 1 -- 172.21.15.7:0/1405689095 shutdown_connections 2023-07-20T02:47:35.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:35.659+0000 7f5f10c00700 1 --2- 172.21.15.7:0/1405689095 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5f0c1012a0 0x7f5f0c108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:47:35.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:35.659+0000 7f5f10c00700 1 --2- 172.21.15.7:0/1405689095 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f0c0ff700 0x7f5f0c10a010 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:47:35.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:35.659+0000 7f5f10c00700 1 --2- 172.21.15.7:0/1405689095 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f5ef40a10c0 0x7f5ef40a3580 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:47:35.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:35.659+0000 7f5f10c00700 1 --2- 172.21.15.7:0/1405689095 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f0c100900 0x7f5f0c108660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:47:35.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:35.659+0000 7f5f10c00700 1 -- 172.21.15.7:0/1405689095 >> 172.21.15.7:0/1405689095 conn(0x7f5f0c0fac80 msgr2=0x7f5f0c0fcef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:47:35.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:35.659+0000 7f5f10c00700 1 -- 172.21.15.7:0/1405689095 shutdown_connections 2023-07-20T02:47:35.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:47:35.659+0000 7f5f10c00700 1 -- 172.21.15.7:0/1405689095 wait complete. 2023-07-20T02:47:36.131 DEBUG:teuthology.orchestra.run.smithi007:osd.3> sudo journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.3.service 2023-07-20T02:47:36.135 INFO:tasks.cephadm:Deploying osd.4 on smithi158 with /dev/vg_nvme/lv_3... 2023-07-20T02:47:36.135 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-20T02:47:36.211 INFO:journalctl@ceph.osd.3.smithi007.stdout:-- Logs begin at Thu 2023-07-20 02:22:32 UTC. -- 2023-07-20T02:47:36.700 INFO:teuthology.orchestra.run.smithi158.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T02:47:36.875 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:36 smithi158 ceph-mon[128925]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:47:36.876 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:36 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:36.876 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:36 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:36.876 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:36 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:47:36.876 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:36 smithi158 ceph-mon[128925]: from='osd.3 [v2:172.21.15.7:6810/2192953827,v1:172.21.15.7:6811/2192953827]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-20T02:47:36.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:36 smithi007 ceph-mon[126353]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:47:36.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:36 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:36.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:36 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:36.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:36 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:47:36.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:36 smithi007 ceph-mon[126353]: from='osd.3 [v2:172.21.15.7:6810/2192953827,v1:172.21.15.7:6811/2192953827]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-20T02:47:36.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:36 smithi188 ceph-mon[129553]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:47:36.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:36 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:36.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:36 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:36.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:36 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:47:36.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:36 smithi188 ceph-mon[129553]: from='osd.3 [v2:172.21.15.7:6810/2192953827,v1:172.21.15.7:6811/2192953827]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-20T02:47:37.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:37 smithi158 ceph-mon[128925]: from='osd.3 [v2:172.21.15.7:6810/2192953827,v1:172.21.15.7:6811/2192953827]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-20T02:47:37.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:37 smithi158 ceph-mon[128925]: osdmap e23: 4 total, 3 up, 4 in 2023-07-20T02:47:37.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:37 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T02:47:37.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:37 smithi158 ceph-mon[128925]: from='osd.3 [v2:172.21.15.7:6810/2192953827,v1:172.21.15.7:6811/2192953827]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-07-20T02:47:37.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:37 smithi007 ceph-mon[126353]: from='osd.3 [v2:172.21.15.7:6810/2192953827,v1:172.21.15.7:6811/2192953827]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-20T02:47:37.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:37 smithi007 ceph-mon[126353]: osdmap e23: 4 total, 3 up, 4 in 2023-07-20T02:47:37.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:37 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T02:47:37.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:37 smithi007 ceph-mon[126353]: from='osd.3 [v2:172.21.15.7:6810/2192953827,v1:172.21.15.7:6811/2192953827]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-07-20T02:47:37.947 INFO:journalctl@ceph.osd.3.smithi007.stdout:Jul 20 02:47:37 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-3[145003]: 2023-07-20T02:47:37.625+0000 7f639daa3700 -1 osd.3 0 waiting for initial osdmap 2023-07-20T02:47:37.947 INFO:journalctl@ceph.osd.3.smithi007.stdout:Jul 20 02:47:37 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-3[145003]: 2023-07-20T02:47:37.633+0000 7f6396092700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T02:47:37.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:37 smithi188 ceph-mon[129553]: from='osd.3 [v2:172.21.15.7:6810/2192953827,v1:172.21.15.7:6811/2192953827]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-20T02:47:37.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:37 smithi188 ceph-mon[129553]: osdmap e23: 4 total, 3 up, 4 in 2023-07-20T02:47:37.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:37 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T02:47:37.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:37 smithi188 ceph-mon[129553]: from='osd.3 [v2:172.21.15.7:6810/2192953827,v1:172.21.15.7:6811/2192953827]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-07-20T02:47:38.026 INFO:teuthology.orchestra.run.smithi158.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-20T02:47:38.026 INFO:teuthology.orchestra.run.smithi158.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-20T02:47:38.026 INFO:teuthology.orchestra.run.smithi158.stderr: stderr: 10+0 records in 2023-07-20T02:47:38.027 INFO:teuthology.orchestra.run.smithi158.stderr:10+0 records out 2023-07-20T02:47:38.027 INFO:teuthology.orchestra.run.smithi158.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00828167 s, 1.3 GB/s 2023-07-20T02:47:38.027 INFO:teuthology.orchestra.run.smithi158.stderr:--> Zapping successful for: 2023-07-20T02:47:38.340 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi158:vg_nvme/lv_3 2023-07-20T02:47:38.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:38 smithi158 ceph-mon[128925]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:47:38.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:38 smithi158 ceph-mon[128925]: from='osd.3 [v2:172.21.15.7:6810/2192953827,v1:172.21.15.7:6811/2192953827]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2023-07-20T02:47:38.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:38 smithi158 ceph-mon[128925]: osdmap e24: 4 total, 3 up, 4 in 2023-07-20T02:47:38.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:38 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T02:47:38.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:38 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T02:47:38.892 INFO:teuthology.orchestra.run.smithi158.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T02:47:38.934 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:38 smithi007 ceph-mon[126353]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:47:38.935 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:38 smithi007 ceph-mon[126353]: from='osd.3 [v2:172.21.15.7:6810/2192953827,v1:172.21.15.7:6811/2192953827]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2023-07-20T02:47:38.935 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:38 smithi007 ceph-mon[126353]: osdmap e24: 4 total, 3 up, 4 in 2023-07-20T02:47:38.935 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:38 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T02:47:38.935 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:38 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T02:47:38.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:38 smithi188 ceph-mon[129553]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:47:38.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:38 smithi188 ceph-mon[129553]: from='osd.3 [v2:172.21.15.7:6810/2192953827,v1:172.21.15.7:6811/2192953827]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2023-07-20T02:47:38.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:38 smithi188 ceph-mon[129553]: osdmap e24: 4 total, 3 up, 4 in 2023-07-20T02:47:38.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:38 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T02:47:38.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:38 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T02:47:39.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:39 smithi158 ceph-mon[128925]: purged_snaps scrub starts 2023-07-20T02:47:39.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:39 smithi158 ceph-mon[128925]: purged_snaps scrub ok 2023-07-20T02:47:39.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:39 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T02:47:39.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:39 smithi158 ceph-mon[128925]: osd.3 [v2:172.21.15.7:6810/2192953827,v1:172.21.15.7:6811/2192953827] boot 2023-07-20T02:47:39.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:39 smithi158 ceph-mon[128925]: osdmap e25: 4 total, 4 up, 4 in 2023-07-20T02:47:39.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:39 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T02:47:39.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:39 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:39.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:39 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:39.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:39 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:47:39.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:39 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:47:39.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:39 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:39.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:39 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:47:39.892 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:39 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:47:39.892 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:39 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:39.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.912+0000 7fbbb0303700 1 -- 172.21.15.158:0/3744964575 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbba80ff700 msgr2=0x7fbba80ffb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:47:39.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.912+0000 7fbbb0303700 1 --2- 172.21.15.158:0/3744964575 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbba80ff700 0x7fbba80ffb20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbb980099c0 tx=0x7fbb98009cd0 comp rx=0 tx=0).stop 2023-07-20T02:47:39.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.912+0000 7fbbb0303700 1 -- 172.21.15.158:0/3744964575 shutdown_connections 2023-07-20T02:47:39.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.912+0000 7fbbb0303700 1 --2- 172.21.15.158:0/3744964575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbba81012a0 0x7fbba8107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:47:39.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.912+0000 7fbbb0303700 1 --2- 172.21.15.158:0/3744964575 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbba80ff700 0x7fbba80ffb20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:47:39.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.912+0000 7fbbb0303700 1 --2- 172.21.15.158:0/3744964575 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbba8100900 0x7fbba8100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:47:39.924 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.912+0000 7fbbb0303700 1 -- 172.21.15.158:0/3744964575 >> 172.21.15.158:0/3744964575 conn(0x7fbba80fac80 msgr2=0x7fbba80fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:47:39.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.912+0000 7fbbb0303700 1 -- 172.21.15.158:0/3744964575 shutdown_connections 2023-07-20T02:47:39.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.912+0000 7fbbb0303700 1 -- 172.21.15.158:0/3744964575 wait complete. 2023-07-20T02:47:39.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.913+0000 7fbbb0303700 1 Processor -- start 2023-07-20T02:47:39.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.913+0000 7fbbb0303700 1 -- start start 2023-07-20T02:47:39.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.913+0000 7fbbb0303700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbba80ff700 0x7fbba8196590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:47:39.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.913+0000 7fbbb0303700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbba8100900 0x7fbba8196ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:47:39.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.913+0000 7fbbb0303700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbba81012a0 0x7fbba819db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:47:39.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.913+0000 7fbbb0303700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba819e120 con 0x7fbba80ff700 2023-07-20T02:47:39.927 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.913+0000 7fbbb0303700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba819e260 con 0x7fbba8100900 2023-07-20T02:47:39.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.913+0000 7fbbb0303700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba819e3f0 con 0x7fbba81012a0 2023-07-20T02:47:39.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.913+0000 7fbbae8a0700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbba81012a0 0x7fbba819db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:47:39.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.913+0000 7fbbae8a0700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbba81012a0 0x7fbba819db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.158:47040/0 (socket says 172.21.15.158:47040) 2023-07-20T02:47:39.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.913+0000 7fbbae8a0700 1 -- 172.21.15.158:0/2361840625 learned_addr learned my addr 172.21.15.158:0/2361840625 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-07-20T02:47:39.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.913+0000 7fbbae09f700 1 --2- 172.21.15.158:0/2361840625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbba80ff700 0x7fbba8196590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:47:39.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.913+0000 7fbbad89e700 1 --2- 172.21.15.158:0/2361840625 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbba8100900 0x7fbba8196ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:47:39.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.914+0000 7fbbae8a0700 1 -- 172.21.15.158:0/2361840625 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbba8100900 msgr2=0x7fbba8196ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:47:39.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.914+0000 7fbbae8a0700 1 --2- 172.21.15.158:0/2361840625 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbba8100900 0x7fbba8196ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:47:39.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.914+0000 7fbbae8a0700 1 -- 172.21.15.158:0/2361840625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbba80ff700 msgr2=0x7fbba8196590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:47:39.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.914+0000 7fbbae8a0700 1 --2- 172.21.15.158:0/2361840625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbba80ff700 0x7fbba8196590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:47:39.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.914+0000 7fbbae8a0700 1 -- 172.21.15.158:0/2361840625 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb980095f0 con 0x7fbba81012a0 2023-07-20T02:47:39.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.914+0000 7fbbae8a0700 1 --2- 172.21.15.158:0/2361840625 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbba81012a0 0x7fbba819db50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbba000b9a0 tx=0x7fbba000bcb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:47:39.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.914+0000 7fbb9f7fe700 1 -- 172.21.15.158:0/2361840625 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba000d850 con 0x7fbba81012a0 2023-07-20T02:47:39.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.914+0000 7fbbb0303700 1 -- 172.21.15.158:0/2361840625 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbba819e6d0 con 0x7fbba81012a0 2023-07-20T02:47:39.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.914+0000 7fbb9f7fe700 1 -- 172.21.15.158:0/2361840625 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbba000de70 con 0x7fbba81012a0 2023-07-20T02:47:39.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.914+0000 7fbb9f7fe700 1 -- 172.21.15.158:0/2361840625 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba0017400 con 0x7fbba81012a0 2023-07-20T02:47:39.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.914+0000 7fbbb0303700 1 -- 172.21.15.158:0/2361840625 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbba819ec50 con 0x7fbba81012a0 2023-07-20T02:47:39.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.915+0000 7fbbb0303700 1 -- 172.21.15.158:0/2361840625 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbba810b000 con 0x7fbba81012a0 2023-07-20T02:47:39.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.916+0000 7fbb9f7fe700 1 -- 172.21.15.158:0/2361840625 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fbba000e600 con 0x7fbba81012a0 2023-07-20T02:47:39.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.917+0000 7fbb9f7fe700 1 --2- 172.21.15.158:0/2361840625 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fbb940a1150 0x7fbb940a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:47:39.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.917+0000 7fbb9f7fe700 1 -- 172.21.15.158:0/2361840625 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 3698+0+0 (secure 0 0 0) 0x7fbba0006e40 con 0x7fbba81012a0 2023-07-20T02:47:39.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.917+0000 7fbbae09f700 1 --2- 172.21.15.158:0/2361840625 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fbb940a1150 0x7fbb940a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:47:39.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.918+0000 7fbbae09f700 1 --2- 172.21.15.158:0/2361840625 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fbb940a1150 0x7fbb940a3610 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbb98009f50 tx=0x7fbb9800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:47:39.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:39.921+0000 7fbb9f7fe700 1 -- 172.21.15.158:0/2361840625 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fbba0095db0 con 0x7fbba81012a0 2023-07-20T02:47:39.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:39 smithi007 ceph-mon[126353]: purged_snaps scrub starts 2023-07-20T02:47:39.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:39 smithi007 ceph-mon[126353]: purged_snaps scrub ok 2023-07-20T02:47:39.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:39 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T02:47:39.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:39 smithi007 ceph-mon[126353]: osd.3 [v2:172.21.15.7:6810/2192953827,v1:172.21.15.7:6811/2192953827] boot 2023-07-20T02:47:39.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:39 smithi007 ceph-mon[126353]: osdmap e25: 4 total, 4 up, 4 in 2023-07-20T02:47:39.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:39 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T02:47:39.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:39 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:39.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:39 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:39.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:39 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:47:39.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:39 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:47:39.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:39 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:39.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:39 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:47:39.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:39 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:47:39.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:39 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:39.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:39 smithi188 ceph-mon[129553]: purged_snaps scrub starts 2023-07-20T02:47:39.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:39 smithi188 ceph-mon[129553]: purged_snaps scrub ok 2023-07-20T02:47:39.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:39 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T02:47:39.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:39 smithi188 ceph-mon[129553]: osd.3 [v2:172.21.15.7:6810/2192953827,v1:172.21.15.7:6811/2192953827] boot 2023-07-20T02:47:39.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:39 smithi188 ceph-mon[129553]: osdmap e25: 4 total, 4 up, 4 in 2023-07-20T02:47:39.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:39 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T02:47:39.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:39 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:39.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:39 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:39.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:39 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:47:39.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:39 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:47:39.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:39 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:39.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:39 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:47:39.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:39 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:47:39.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:39 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:40.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:40.112+0000 7fbbb0303700 1 -- 172.21.15.158:0/2361840625 --> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi158:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fbba819f2f0 con 0x7fbb940a1150 2023-07-20T02:47:40.915 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:40 smithi158 ceph-mon[128925]: Detected new or changed devices on smithi007 2023-07-20T02:47:40.916 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:40 smithi158 ceph-mon[128925]: Adjusting osd_memory_target on smithi007 to 8636M 2023-07-20T02:47:40.916 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:40 smithi158 ceph-mon[128925]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:47:40.916 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:40 smithi158 ceph-mon[128925]: osdmap e26: 4 total, 4 up, 4 in 2023-07-20T02:47:40.916 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:40 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:47:40.916 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:40 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:47:40.917 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:40 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:47:40.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:40 smithi007 ceph-mon[126353]: Detected new or changed devices on smithi007 2023-07-20T02:47:40.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:40 smithi007 ceph-mon[126353]: Adjusting osd_memory_target on smithi007 to 8636M 2023-07-20T02:47:40.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:40 smithi007 ceph-mon[126353]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:47:40.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:40 smithi007 ceph-mon[126353]: osdmap e26: 4 total, 4 up, 4 in 2023-07-20T02:47:40.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:40 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:47:40.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:40 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:47:40.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:40 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:47:40.960 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:40 smithi188 ceph-mon[129553]: Detected new or changed devices on smithi007 2023-07-20T02:47:40.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:40 smithi188 ceph-mon[129553]: Adjusting osd_memory_target on smithi007 to 8636M 2023-07-20T02:47:40.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:40 smithi188 ceph-mon[129553]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-20T02:47:40.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:40 smithi188 ceph-mon[129553]: osdmap e26: 4 total, 4 up, 4 in 2023-07-20T02:47:40.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:40 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:47:40.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:40 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:47:40.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:40 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:47:41.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:41 smithi158 ceph-mon[128925]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi158:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:47:41.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:41 smithi158 ceph-mon[128925]: osdmap e27: 4 total, 4 up, 4 in 2023-07-20T02:47:41.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:41 smithi007 ceph-mon[126353]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi158:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:47:41.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:41 smithi007 ceph-mon[126353]: osdmap e27: 4 total, 4 up, 4 in 2023-07-20T02:47:41.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:41 smithi188 ceph-mon[129553]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi158:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:47:41.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:41 smithi188 ceph-mon[129553]: osdmap e27: 4 total, 4 up, 4 in 2023-07-20T02:47:42.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:42 smithi007 ceph-mon[126353]: pgmap v101: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-20T02:47:42.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:42 smithi007 ceph-mon[126353]: osdmap e28: 4 total, 4 up, 4 in 2023-07-20T02:47:42.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:42 smithi007 ceph-mon[126353]: from='client.? 172.21.15.158:0/2896042703' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2453384-3b09-4f2a-bea7-172d8fc466d6"}]: dispatch 2023-07-20T02:47:42.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:42 smithi007 ceph-mon[126353]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2453384-3b09-4f2a-bea7-172d8fc466d6"}]: dispatch 2023-07-20T02:47:42.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:42 smithi007 ceph-mon[126353]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e2453384-3b09-4f2a-bea7-172d8fc466d6"}]': finished 2023-07-20T02:47:42.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:42 smithi007 ceph-mon[126353]: osdmap e29: 5 total, 4 up, 5 in 2023-07-20T02:47:42.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:42 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T02:47:42.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:42 smithi188 ceph-mon[129553]: pgmap v101: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-20T02:47:42.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:42 smithi188 ceph-mon[129553]: osdmap e28: 4 total, 4 up, 4 in 2023-07-20T02:47:42.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:42 smithi188 ceph-mon[129553]: from='client.? 172.21.15.158:0/2896042703' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2453384-3b09-4f2a-bea7-172d8fc466d6"}]: dispatch 2023-07-20T02:47:42.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:42 smithi188 ceph-mon[129553]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2453384-3b09-4f2a-bea7-172d8fc466d6"}]: dispatch 2023-07-20T02:47:42.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:42 smithi188 ceph-mon[129553]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e2453384-3b09-4f2a-bea7-172d8fc466d6"}]': finished 2023-07-20T02:47:42.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:42 smithi188 ceph-mon[129553]: osdmap e29: 5 total, 4 up, 5 in 2023-07-20T02:47:42.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:42 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T02:47:43.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:42 smithi158 ceph-mon[128925]: pgmap v101: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-20T02:47:43.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:42 smithi158 ceph-mon[128925]: osdmap e28: 4 total, 4 up, 4 in 2023-07-20T02:47:43.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:42 smithi158 ceph-mon[128925]: from='client.? 172.21.15.158:0/2896042703' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2453384-3b09-4f2a-bea7-172d8fc466d6"}]: dispatch 2023-07-20T02:47:43.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:42 smithi158 ceph-mon[128925]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2453384-3b09-4f2a-bea7-172d8fc466d6"}]: dispatch 2023-07-20T02:47:43.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:42 smithi158 ceph-mon[128925]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e2453384-3b09-4f2a-bea7-172d8fc466d6"}]': finished 2023-07-20T02:47:43.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:42 smithi158 ceph-mon[128925]: osdmap e29: 5 total, 4 up, 5 in 2023-07-20T02:47:43.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:42 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T02:47:43.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:43 smithi007 ceph-mon[126353]: from='client.? 172.21.15.158:0/31123006' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:47:43.957 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:43 smithi007 ceph-mon[126353]: pgmap v104: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-20T02:47:43.965 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:43 smithi188 ceph-mon[129553]: from='client.? 172.21.15.158:0/31123006' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:47:43.965 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:43 smithi188 ceph-mon[129553]: pgmap v104: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-20T02:47:44.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:43 smithi158 ceph-mon[128925]: from='client.? 172.21.15.158:0/31123006' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:47:44.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:43 smithi158 ceph-mon[128925]: pgmap v104: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-20T02:47:46.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:46 smithi158 ceph-mon[128925]: pgmap v105: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2023-07-20T02:47:46.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:46 smithi007 ceph-mon[126353]: pgmap v105: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2023-07-20T02:47:46.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:46 smithi188 ceph-mon[129553]: pgmap v105: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2023-07-20T02:47:48.367 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:48 smithi158 ceph-mon[128925]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 84 KiB/s, 0 objects/s recovering 2023-07-20T02:47:48.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:48 smithi007 ceph-mon[126353]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 84 KiB/s, 0 objects/s recovering 2023-07-20T02:47:48.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:48 smithi188 ceph-mon[129553]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 84 KiB/s, 0 objects/s recovering 2023-07-20T02:47:49.387 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:49 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-20T02:47:49.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:49 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:47:49.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:49 smithi158 ceph-mon[128925]: Deploying daemon osd.4 on smithi158 2023-07-20T02:47:49.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:49 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-20T02:47:49.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:49 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:47:49.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:49 smithi007 ceph-mon[126353]: Deploying daemon osd.4 on smithi158 2023-07-20T02:47:49.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:49 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-20T02:47:49.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:49 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:47:49.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:49 smithi188 ceph-mon[129553]: Deploying daemon osd.4 on smithi158 2023-07-20T02:47:50.375 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:50 smithi158 ceph-mon[128925]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:47:50.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:50 smithi007 ceph-mon[126353]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:47:50.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:50 smithi188 ceph-mon[129553]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:47:52.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:52 smithi158 ceph-mon[128925]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:47:52.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:52 smithi007 ceph-mon[126353]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:47:52.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:52 smithi188 ceph-mon[129553]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:47:53.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:53 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:47:53.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:53 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:53.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:53 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:53.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:53 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:47:53.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:53 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:53.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:53 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:53.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:53 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:47:53.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:53 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:53.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:53 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:54.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:54 smithi158 ceph-mon[128925]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:47:54.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:54 smithi007 ceph-mon[126353]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:47:54.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:54 smithi188 ceph-mon[129553]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:47:55.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:55.326+0000 7fbb9f7fe700 1 -- 172.21.15.158:0/2361840625 <== mgr.14152 v2:172.21.15.7:6800/1354542137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fbba819f2f0 con 0x7fbb940a1150 2023-07-20T02:47:55.337 INFO:teuthology.orchestra.run.smithi158.stdout:Created osd(s) 4 on host 'smithi158' 2023-07-20T02:47:55.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:55.328+0000 7fbbb0303700 1 -- 172.21.15.158:0/2361840625 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fbb940a1150 msgr2=0x7fbb940a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:47:55.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:55.328+0000 7fbbb0303700 1 --2- 172.21.15.158:0/2361840625 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fbb940a1150 0x7fbb940a3610 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbb98009f50 tx=0x7fbb9800b540 comp rx=0 tx=0).stop 2023-07-20T02:47:55.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:55.328+0000 7fbbb0303700 1 -- 172.21.15.158:0/2361840625 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbba81012a0 msgr2=0x7fbba819db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:47:55.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:55.328+0000 7fbbb0303700 1 --2- 172.21.15.158:0/2361840625 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbba81012a0 0x7fbba819db50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbba000b9a0 tx=0x7fbba000bcb0 comp rx=0 tx=0).stop 2023-07-20T02:47:55.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:55.328+0000 7fbbb0303700 1 -- 172.21.15.158:0/2361840625 shutdown_connections 2023-07-20T02:47:55.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:55.328+0000 7fbbb0303700 1 --2- 172.21.15.158:0/2361840625 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbba81012a0 0x7fbba819db50 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:47:55.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:55.328+0000 7fbbb0303700 1 --2- 172.21.15.158:0/2361840625 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fbb940a1150 0x7fbb940a3610 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:47:55.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:55.328+0000 7fbbb0303700 1 --2- 172.21.15.158:0/2361840625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbba80ff700 0x7fbba8196590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:47:55.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:55.328+0000 7fbbb0303700 1 --2- 172.21.15.158:0/2361840625 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbba8100900 0x7fbba8196ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:47:55.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:55.328+0000 7fbbb0303700 1 -- 172.21.15.158:0/2361840625 >> 172.21.15.158:0/2361840625 conn(0x7fbba80fac80 msgr2=0x7fbba8068250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:47:55.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:55.328+0000 7fbbb0303700 1 -- 172.21.15.158:0/2361840625 shutdown_connections 2023-07-20T02:47:55.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:47:55.328+0000 7fbbb0303700 1 -- 172.21.15.158:0/2361840625 wait complete. 2023-07-20T02:47:55.769 DEBUG:teuthology.orchestra.run.smithi158:osd.4> sudo journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.4.service 2023-07-20T02:47:55.771 INFO:tasks.cephadm:Deploying osd.5 on smithi188 with /dev/vg_nvme/lv_3... 2023-07-20T02:47:55.772 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-20T02:47:55.819 INFO:journalctl@ceph.osd.4.smithi158.stdout:-- Logs begin at Thu 2023-07-20 02:10:12 UTC. -- 2023-07-20T02:47:56.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:55 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:56.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:55 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:56.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:55 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:47:56.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:55 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:47:56.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:55 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:56.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:55 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:56.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:55 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:56.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:55 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:47:56.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:55 smithi158 ceph-mon[128925]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:47:56.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:55 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:56.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:55 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:56.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:55 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:47:56.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:55 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:47:56.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:55 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:56.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:55 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:56.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:55 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:56.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:55 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:47:56.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:55 smithi007 ceph-mon[126353]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:47:56.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:55 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:56.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:55 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:56.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:55 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:47:56.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:55 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:47:56.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:55 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:56.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:55 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:56.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:55 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:56.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:55 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:47:56.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:55 smithi188 ceph-mon[129553]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:47:56.346 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T02:47:56.888 INFO:journalctl@ceph.osd.4.smithi158.stdout:Jul 20 02:47:56 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-4[139474]: 2023-07-20T02:47:56.648+0000 7f2e58522640 -1 osd.4 0 log_to_monitors true 2023-07-20T02:47:57.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:57 smithi188 ceph-mon[129553]: from='osd.4 [v2:172.21.15.158:6808/3330591132,v1:172.21.15.158:6809/3330591132]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T02:47:57.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:57 smithi188 ceph-mon[129553]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T02:47:57.577 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:57 smithi158 ceph-mon[128925]: from='osd.4 [v2:172.21.15.158:6808/3330591132,v1:172.21.15.158:6809/3330591132]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T02:47:57.578 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:57 smithi158 ceph-mon[128925]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T02:47:57.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:57 smithi007 ceph-mon[126353]: from='osd.4 [v2:172.21.15.158:6808/3330591132,v1:172.21.15.158:6809/3330591132]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T02:47:57.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:57 smithi007 ceph-mon[126353]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T02:47:57.995 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-20T02:47:57.996 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-20T02:47:57.996 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2023-07-20T02:47:57.996 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2023-07-20T02:47:57.996 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109227 s, 960 MB/s 2023-07-20T02:47:57.996 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2023-07-20T02:47:58.400 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi188:vg_nvme/lv_3 2023-07-20T02:47:58.638 INFO:journalctl@ceph.osd.4.smithi158.stdout:Jul 20 02:47:58 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-4[139474]: 2023-07-20T02:47:58.301+0000 7f2e4db9a700 -1 osd.4 0 waiting for initial osdmap 2023-07-20T02:47:58.638 INFO:journalctl@ceph.osd.4.smithi158.stdout:Jul 20 02:47:58 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-4[139474]: 2023-07-20T02:47:58.307+0000 7f2e4718b700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T02:47:58.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:58 smithi158 ceph-mon[128925]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-20T02:47:58.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:58 smithi158 ceph-mon[128925]: osdmap e30: 5 total, 4 up, 5 in 2023-07-20T02:47:58.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:58 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T02:47:58.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:58 smithi158 ceph-mon[128925]: from='osd.4 [v2:172.21.15.158:6808/3330591132,v1:172.21.15.158:6809/3330591132]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]: dispatch 2023-07-20T02:47:58.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:58 smithi158 ceph-mon[128925]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]: dispatch 2023-07-20T02:47:58.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:58 smithi158 ceph-mon[128925]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:47:58.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:58 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:58.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:58 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:58.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:58 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:47:58.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:58 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:47:58.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:58 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:58.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:58 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:47:58.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:58 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:47:58.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:58 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:58.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:58 smithi007 ceph-mon[126353]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-20T02:47:58.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:58 smithi007 ceph-mon[126353]: osdmap e30: 5 total, 4 up, 5 in 2023-07-20T02:47:58.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:58 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T02:47:58.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:58 smithi007 ceph-mon[126353]: from='osd.4 [v2:172.21.15.158:6808/3330591132,v1:172.21.15.158:6809/3330591132]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]: dispatch 2023-07-20T02:47:58.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:58 smithi007 ceph-mon[126353]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]: dispatch 2023-07-20T02:47:58.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:58 smithi007 ceph-mon[126353]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:47:58.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:58 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:58.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:58 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:58.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:58 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:47:58.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:58 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:47:58.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:58 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:58.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:58 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:47:58.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:58 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:47:58.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:58 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:58.711 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:58 smithi188 ceph-mon[129553]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-20T02:47:58.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:58 smithi188 ceph-mon[129553]: osdmap e30: 5 total, 4 up, 5 in 2023-07-20T02:47:58.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:58 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T02:47:58.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:58 smithi188 ceph-mon[129553]: from='osd.4 [v2:172.21.15.158:6808/3330591132,v1:172.21.15.158:6809/3330591132]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]: dispatch 2023-07-20T02:47:58.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:58 smithi188 ceph-mon[129553]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]: dispatch 2023-07-20T02:47:58.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:58 smithi188 ceph-mon[129553]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:47:58.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:58 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:58.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:58 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:58.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:58 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:47:58.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:58 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:47:58.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:58 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:58.714 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:58 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:47:58.714 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:58 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:47:58.714 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:58 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:47:58.952 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T02:47:59.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:59 smithi188 ceph-mon[129553]: Detected new or changed devices on smithi158 2023-07-20T02:47:59.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:59 smithi188 ceph-mon[129553]: Adjusting osd_memory_target on smithi158 to 8584M 2023-07-20T02:47:59.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:59 smithi188 ceph-mon[129553]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]': finished 2023-07-20T02:47:59.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:59 smithi188 ceph-mon[129553]: osdmap e31: 5 total, 4 up, 5 in 2023-07-20T02:47:59.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:59 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T02:47:59.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:47:59 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T02:47:59.638 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:59 smithi158 ceph-mon[128925]: Detected new or changed devices on smithi158 2023-07-20T02:47:59.638 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:59 smithi158 ceph-mon[128925]: Adjusting osd_memory_target on smithi158 to 8584M 2023-07-20T02:47:59.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:59 smithi158 ceph-mon[128925]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]': finished 2023-07-20T02:47:59.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:59 smithi158 ceph-mon[128925]: osdmap e31: 5 total, 4 up, 5 in 2023-07-20T02:47:59.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:59 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T02:47:59.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:47:59 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T02:47:59.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:59 smithi007 ceph-mon[126353]: Detected new or changed devices on smithi158 2023-07-20T02:47:59.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:59 smithi007 ceph-mon[126353]: Adjusting osd_memory_target on smithi158 to 8584M 2023-07-20T02:47:59.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:59 smithi007 ceph-mon[126353]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]': finished 2023-07-20T02:47:59.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:59 smithi007 ceph-mon[126353]: osdmap e31: 5 total, 4 up, 5 in 2023-07-20T02:47:59.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:59 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T02:47:59.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:47:59 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T02:48:00.301 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.290+0000 7f3b6abd5700 1 -- 172.21.15.188:0/2692426446 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b641012a0 msgr2=0x7f3b64107b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:48:00.302 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.290+0000 7f3b6abd5700 1 --2- 172.21.15.188:0/2692426446 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b641012a0 0x7f3b64107b50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c00b0a0 tx=0x7f3b5c00b3b0 comp rx=0 tx=0).stop 2023-07-20T02:48:00.302 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.290+0000 7f3b6abd5700 1 -- 172.21.15.188:0/2692426446 shutdown_connections 2023-07-20T02:48:00.302 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.290+0000 7f3b6abd5700 1 --2- 172.21.15.188:0/2692426446 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b641012a0 0x7f3b64107b50 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:48:00.302 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.290+0000 7f3b6abd5700 1 --2- 172.21.15.188:0/2692426446 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b640ff700 0x7f3b640ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:48:00.302 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.290+0000 7f3b6abd5700 1 --2- 172.21.15.188:0/2692426446 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b64100900 0x7f3b64100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:48:00.302 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.290+0000 7f3b6abd5700 1 -- 172.21.15.188:0/2692426446 >> 172.21.15.188:0/2692426446 conn(0x7f3b640fac80 msgr2=0x7f3b640fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:48:00.302 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.290+0000 7f3b6abd5700 1 -- 172.21.15.188:0/2692426446 shutdown_connections 2023-07-20T02:48:00.303 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.290+0000 7f3b6abd5700 1 -- 172.21.15.188:0/2692426446 wait complete. 2023-07-20T02:48:00.304 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.291+0000 7f3b6abd5700 1 Processor -- start 2023-07-20T02:48:00.305 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.291+0000 7f3b6abd5700 1 -- start start 2023-07-20T02:48:00.305 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.291+0000 7f3b6abd5700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b640ff700 0x7f3b64195520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:48:00.305 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.291+0000 7f3b6abd5700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b64100900 0x7f3b64195a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:48:00.305 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.291+0000 7f3b6abd5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b641012a0 0x7f3b6419cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:48:00.305 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.291+0000 7f3b6abd5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6419d0b0 con 0x7f3b641012a0 2023-07-20T02:48:00.305 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.291+0000 7f3b6abd5700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6419d1f0 con 0x7f3b640ff700 2023-07-20T02:48:00.305 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.291+0000 7f3b6abd5700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6419d380 con 0x7f3b64100900 2023-07-20T02:48:00.306 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.291+0000 7f3b63fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b64100900 0x7f3b64195a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:48:00.306 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.291+0000 7f3b63fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b64100900 0x7f3b64195a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:34936/0 (socket says 172.21.15.188:34936) 2023-07-20T02:48:00.306 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.291+0000 7f3b63fff700 1 -- 172.21.15.188:0/1942613041 learned_addr learned my addr 172.21.15.188:0/1942613041 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-07-20T02:48:00.306 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.292+0000 7f3b69172700 1 --2- 172.21.15.188:0/1942613041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b641012a0 0x7f3b6419cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:48:00.306 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.292+0000 7f3b68971700 1 --2- 172.21.15.188:0/1942613041 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b640ff700 0x7f3b64195520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:48:00.306 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.292+0000 7f3b63fff700 1 -- 172.21.15.188:0/1942613041 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b640ff700 msgr2=0x7f3b64195520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:48:00.306 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.292+0000 7f3b63fff700 1 --2- 172.21.15.188:0/1942613041 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b640ff700 0x7f3b64195520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:48:00.307 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.292+0000 7f3b63fff700 1 -- 172.21.15.188:0/1942613041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b641012a0 msgr2=0x7f3b6419cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:48:00.307 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.292+0000 7f3b63fff700 1 --2- 172.21.15.188:0/1942613041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b641012a0 0x7f3b6419cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:48:00.307 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.292+0000 7f3b63fff700 1 -- 172.21.15.188:0/1942613041 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b5c009d00 con 0x7f3b64100900 2023-07-20T02:48:00.307 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.292+0000 7f3b63fff700 1 --2- 172.21.15.188:0/1942613041 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b64100900 0x7f3b64195a60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3b5000c950 tx=0x7f3b5000cd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:48:00.307 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.292+0000 7f3b61ffb700 1 -- 172.21.15.188:0/1942613041 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b50007ac0 con 0x7f3b64100900 2023-07-20T02:48:00.307 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.292+0000 7f3b61ffb700 1 -- 172.21.15.188:0/1942613041 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b50004cf0 con 0x7f3b64100900 2023-07-20T02:48:00.307 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.292+0000 7f3b6abd5700 1 -- 172.21.15.188:0/1942613041 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b6419d660 con 0x7f3b64100900 2023-07-20T02:48:00.308 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.292+0000 7f3b61ffb700 1 -- 172.21.15.188:0/1942613041 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b50005550 con 0x7f3b64100900 2023-07-20T02:48:00.308 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.292+0000 7f3b6abd5700 1 -- 172.21.15.188:0/1942613041 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b6419dbe0 con 0x7f3b64100900 2023-07-20T02:48:00.308 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.293+0000 7f3b6abd5700 1 -- 172.21.15.188:0/1942613041 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b6410afa0 con 0x7f3b64100900 2023-07-20T02:48:00.308 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.294+0000 7f3b61ffb700 1 -- 172.21.15.188:0/1942613041 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f3b50004750 con 0x7f3b64100900 2023-07-20T02:48:00.308 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.295+0000 7f3b61ffb700 1 --2- 172.21.15.188:0/1942613041 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f3b540a1150 0x7f3b540a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:48:00.309 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.295+0000 7f3b61ffb700 1 -- 172.21.15.188:0/1942613041 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f3b500cb4a0 con 0x7f3b64100900 2023-07-20T02:48:00.309 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.295+0000 7f3b68971700 1 --2- 172.21.15.188:0/1942613041 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f3b540a1150 0x7f3b540a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:48:00.309 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.296+0000 7f3b68971700 1 --2- 172.21.15.188:0/1942613041 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f3b540a1150 0x7f3b540a3610 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3b58006fd0 tx=0x7f3b58008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:48:00.309 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.299+0000 7f3b61ffb700 1 -- 172.21.15.188:0/1942613041 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3b50095c60 con 0x7f3b64100900 2023-07-20T02:48:00.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:00 smithi188 ceph-mon[129553]: purged_snaps scrub starts 2023-07-20T02:48:00.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:00 smithi188 ceph-mon[129553]: purged_snaps scrub ok 2023-07-20T02:48:00.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:00 smithi188 ceph-mon[129553]: osd.4 [v2:172.21.15.158:6808/3330591132,v1:172.21.15.158:6809/3330591132] boot 2023-07-20T02:48:00.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:00 smithi188 ceph-mon[129553]: osdmap e32: 5 total, 5 up, 5 in 2023-07-20T02:48:00.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:00 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T02:48:00.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:00 smithi188 ceph-mon[129553]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:48:00.504 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:00.494+0000 7f3b6abd5700 1 -- 172.21.15.188:0/1942613041 --> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f3b64060bb0 con 0x7f3b540a1150 2023-07-20T02:48:00.638 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:00 smithi158 ceph-mon[128925]: purged_snaps scrub starts 2023-07-20T02:48:00.638 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:00 smithi158 ceph-mon[128925]: purged_snaps scrub ok 2023-07-20T02:48:00.638 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:00 smithi158 ceph-mon[128925]: osd.4 [v2:172.21.15.158:6808/3330591132,v1:172.21.15.158:6809/3330591132] boot 2023-07-20T02:48:00.638 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:00 smithi158 ceph-mon[128925]: osdmap e32: 5 total, 5 up, 5 in 2023-07-20T02:48:00.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:00 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T02:48:00.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:00 smithi158 ceph-mon[128925]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:48:00.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:00 smithi007 ceph-mon[126353]: purged_snaps scrub starts 2023-07-20T02:48:00.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:00 smithi007 ceph-mon[126353]: purged_snaps scrub ok 2023-07-20T02:48:00.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:00 smithi007 ceph-mon[126353]: osd.4 [v2:172.21.15.158:6808/3330591132,v1:172.21.15.158:6809/3330591132] boot 2023-07-20T02:48:00.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:00 smithi007 ceph-mon[126353]: osdmap e32: 5 total, 5 up, 5 in 2023-07-20T02:48:00.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:00 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T02:48:00.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:00 smithi007 ceph-mon[126353]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-20T02:48:01.638 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:01 smithi158 ceph-mon[128925]: osdmap e33: 5 total, 5 up, 5 in 2023-07-20T02:48:01.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:01 smithi158 ceph-mon[128925]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:48:01.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:01 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:48:01.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:01 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:48:01.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:01 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:48:01.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:01 smithi188 ceph-mon[129553]: osdmap e33: 5 total, 5 up, 5 in 2023-07-20T02:48:01.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:01 smithi188 ceph-mon[129553]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:48:01.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:01 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:48:01.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:01 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:48:01.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:01 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:48:01.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:01 smithi007 ceph-mon[126353]: osdmap e33: 5 total, 5 up, 5 in 2023-07-20T02:48:01.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:01 smithi007 ceph-mon[126353]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:48:01.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:01 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:48:01.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:01 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:48:01.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:01 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:48:02.638 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:02 smithi158 ceph-mon[128925]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:48:02.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:02 smithi007 ceph-mon[126353]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:48:02.711 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:02 smithi188 ceph-mon[129553]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:48:03.638 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:03 smithi158 ceph-mon[128925]: from='client.? 172.21.15.188:0/3187571522' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3c04c13-0ce5-440d-9e62-0e301ad9f771"}]: dispatch 2023-07-20T02:48:03.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:03 smithi158 ceph-mon[128925]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3c04c13-0ce5-440d-9e62-0e301ad9f771"}]: dispatch 2023-07-20T02:48:03.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:03 smithi158 ceph-mon[128925]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3c04c13-0ce5-440d-9e62-0e301ad9f771"}]': finished 2023-07-20T02:48:03.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:03 smithi158 ceph-mon[128925]: osdmap e34: 6 total, 5 up, 6 in 2023-07-20T02:48:03.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:03 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T02:48:03.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:03 smithi007 ceph-mon[126353]: from='client.? 172.21.15.188:0/3187571522' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3c04c13-0ce5-440d-9e62-0e301ad9f771"}]: dispatch 2023-07-20T02:48:03.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:03 smithi007 ceph-mon[126353]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3c04c13-0ce5-440d-9e62-0e301ad9f771"}]: dispatch 2023-07-20T02:48:03.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:03 smithi007 ceph-mon[126353]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3c04c13-0ce5-440d-9e62-0e301ad9f771"}]': finished 2023-07-20T02:48:03.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:03 smithi007 ceph-mon[126353]: osdmap e34: 6 total, 5 up, 6 in 2023-07-20T02:48:03.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:03 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T02:48:03.711 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:03 smithi188 ceph-mon[129553]: from='client.? 172.21.15.188:0/3187571522' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3c04c13-0ce5-440d-9e62-0e301ad9f771"}]: dispatch 2023-07-20T02:48:03.711 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:03 smithi188 ceph-mon[129553]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3c04c13-0ce5-440d-9e62-0e301ad9f771"}]: dispatch 2023-07-20T02:48:03.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:03 smithi188 ceph-mon[129553]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3c04c13-0ce5-440d-9e62-0e301ad9f771"}]': finished 2023-07-20T02:48:03.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:03 smithi188 ceph-mon[129553]: osdmap e34: 6 total, 5 up, 6 in 2023-07-20T02:48:03.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:03 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T02:48:04.638 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:04 smithi158 ceph-mon[128925]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:48:04.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:04 smithi158 ceph-mon[128925]: from='client.? 172.21.15.188:0/974533002' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:48:04.695 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:04 smithi007 ceph-mon[126353]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:48:04.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:04 smithi007 ceph-mon[126353]: from='client.? 172.21.15.188:0/974533002' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:48:04.711 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:04 smithi188 ceph-mon[129553]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:48:04.711 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:04 smithi188 ceph-mon[129553]: from='client.? 172.21.15.188:0/974533002' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:48:06.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:06 smithi158 ceph-mon[128925]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:48:06.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:06 smithi007 ceph-mon[126353]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:48:06.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:06 smithi188 ceph-mon[129553]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:48:08.209 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:08 smithi188 ceph-mon[129553]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:48:08.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:08 smithi158 ceph-mon[128925]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:48:08.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:08 smithi007 ceph-mon[126353]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:48:10.843 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:10 smithi188 ceph-mon[129553]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:48:10.843 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:10 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-20T02:48:10.843 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:10 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:48:10.844 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:10 smithi188 ceph-mon[129553]: Deploying daemon osd.5 on smithi188 2023-07-20T02:48:10.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:10 smithi158 ceph-mon[128925]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:48:10.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:10 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-20T02:48:10.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:10 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:48:10.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:10 smithi158 ceph-mon[128925]: Deploying daemon osd.5 on smithi188 2023-07-20T02:48:10.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:10 smithi007 ceph-mon[126353]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:48:10.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:10 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-20T02:48:10.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:10 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:48:10.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:10 smithi007 ceph-mon[126353]: Deploying daemon osd.5 on smithi188 2023-07-20T02:48:12.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:12 smithi158 ceph-mon[128925]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:48:12.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:12 smithi007 ceph-mon[126353]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:48:12.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:12 smithi188 ceph-mon[129553]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:48:14.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:14 smithi158 ceph-mon[128925]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:48:14.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:14 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:14.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:14 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:14.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:14 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:48:14.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:14 smithi007 ceph-mon[126353]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:48:14.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:14 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:14.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:14 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:14.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:14 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:48:14.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:14 smithi188 ceph-mon[129553]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:48:14.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:14 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:14.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:14 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:14.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:14 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:48:16.829 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:16 smithi188 ceph-mon[129553]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:48:16.829 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:16 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:16.829 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:16 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:16.829 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:16 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:48:16.829 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:16 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:48:16.829 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:16 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:16.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:16 smithi158 ceph-mon[128925]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:48:16.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:16 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:16.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:16 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:16.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:16 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:48:16.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:16 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:48:16.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:16 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:16.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:16 smithi007 ceph-mon[126353]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:48:16.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:16 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:16.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:16 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:16.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:16 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:48:16.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:16 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:48:16.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:16 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:17.277 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:17.266+0000 7f3b61ffb700 1 -- 172.21.15.188:0/1942613041 <== mgr.14152 v2:172.21.15.7:6800/1354542137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3b64060bb0 con 0x7f3b540a1150 2023-07-20T02:48:17.278 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 5 on host 'smithi188' 2023-07-20T02:48:17.279 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:17.268+0000 7f3b6abd5700 1 -- 172.21.15.188:0/1942613041 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f3b540a1150 msgr2=0x7f3b540a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:48:17.280 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:17.268+0000 7f3b6abd5700 1 --2- 172.21.15.188:0/1942613041 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f3b540a1150 0x7f3b540a3610 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3b58006fd0 tx=0x7f3b58008040 comp rx=0 tx=0).stop 2023-07-20T02:48:17.280 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:17.268+0000 7f3b6abd5700 1 -- 172.21.15.188:0/1942613041 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b64100900 msgr2=0x7f3b64195a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:48:17.280 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:17.268+0000 7f3b6abd5700 1 --2- 172.21.15.188:0/1942613041 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b64100900 0x7f3b64195a60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3b5000c950 tx=0x7f3b5000cd10 comp rx=0 tx=0).stop 2023-07-20T02:48:17.280 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:17.268+0000 7f3b6abd5700 1 -- 172.21.15.188:0/1942613041 shutdown_connections 2023-07-20T02:48:17.280 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:17.268+0000 7f3b6abd5700 1 --2- 172.21.15.188:0/1942613041 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f3b540a1150 0x7f3b540a3610 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:48:17.280 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:17.268+0000 7f3b6abd5700 1 --2- 172.21.15.188:0/1942613041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b641012a0 0x7f3b6419cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:48:17.281 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:17.268+0000 7f3b6abd5700 1 --2- 172.21.15.188:0/1942613041 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3b640ff700 0x7f3b64195520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:48:17.281 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:17.268+0000 7f3b6abd5700 1 --2- 172.21.15.188:0/1942613041 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b64100900 0x7f3b64195a60 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:48:17.281 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:17.268+0000 7f3b6abd5700 1 -- 172.21.15.188:0/1942613041 >> 172.21.15.188:0/1942613041 conn(0x7f3b640fac80 msgr2=0x7f3b640fce80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:48:17.281 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:17.268+0000 7f3b6abd5700 1 -- 172.21.15.188:0/1942613041 shutdown_connections 2023-07-20T02:48:17.281 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:48:17.269+0000 7f3b6abd5700 1 -- 172.21.15.188:0/1942613041 wait complete. 2023-07-20T02:48:17.647 DEBUG:teuthology.orchestra.run.smithi188:osd.5> sudo journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.5.service 2023-07-20T02:48:17.649 INFO:tasks.cephadm:Deploying osd.6 on smithi007 with /dev/vg_nvme/lv_2... 2023-07-20T02:48:17.650 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-20T02:48:17.679 INFO:journalctl@ceph.osd.5.smithi188.stdout:-- Logs begin at Thu 2023-07-20 02:09:43 UTC. -- 2023-07-20T02:48:18.413 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:48:18.430 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:48:18 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5[140890]: 2023-07-20T02:48:18.151+0000 7fb76585e640 -1 osd.5 0 log_to_monitors true 2023-07-20T02:48:18.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:18 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:18.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:18 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:18.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:18 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:48:18.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:18 smithi188 ceph-mon[129553]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:48:18.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:18 smithi188 ceph-mon[129553]: from='osd.5 [v2:172.21.15.188:6808/3142012805,v1:172.21.15.188:6809/3142012805]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T02:48:18.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:18 smithi188 ceph-mon[129553]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T02:48:18.638 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:18 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:18.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:18 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:18.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:18 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:48:18.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:18 smithi158 ceph-mon[128925]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:48:18.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:18 smithi158 ceph-mon[128925]: from='osd.5 [v2:172.21.15.188:6808/3142012805,v1:172.21.15.188:6809/3142012805]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T02:48:18.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:18 smithi158 ceph-mon[128925]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T02:48:18.667 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:18 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:18.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:18 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:18.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:18 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:48:18.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:18 smithi007 ceph-mon[126353]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:48:18.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:18 smithi007 ceph-mon[126353]: from='osd.5 [v2:172.21.15.188:6808/3142012805,v1:172.21.15.188:6809/3142012805]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T02:48:18.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:18 smithi007 ceph-mon[126353]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T02:48:19.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:19 smithi188 ceph-mon[129553]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-20T02:48:19.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:19 smithi188 ceph-mon[129553]: osdmap e35: 6 total, 5 up, 6 in 2023-07-20T02:48:19.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:19 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T02:48:19.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:19 smithi188 ceph-mon[129553]: from='osd.5 [v2:172.21.15.188:6808/3142012805,v1:172.21.15.188:6809/3142012805]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-07-20T02:48:19.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:19 smithi188 ceph-mon[129553]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-07-20T02:48:19.462 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:48:19 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5[140890]: 2023-07-20T02:48:19.258+0000 7fb75aed6700 -1 osd.5 0 waiting for initial osdmap 2023-07-20T02:48:19.463 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:48:19 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5[140890]: 2023-07-20T02:48:19.264+0000 7fb754cc8700 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T02:48:19.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:19 smithi007 ceph-mon[126353]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-20T02:48:19.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:19 smithi007 ceph-mon[126353]: osdmap e35: 6 total, 5 up, 6 in 2023-07-20T02:48:19.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:19 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T02:48:19.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:19 smithi007 ceph-mon[126353]: from='osd.5 [v2:172.21.15.188:6808/3142012805,v1:172.21.15.188:6809/3142012805]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-07-20T02:48:19.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:19 smithi007 ceph-mon[126353]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-07-20T02:48:19.638 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:19 smithi158 ceph-mon[128925]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-20T02:48:19.638 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:19 smithi158 ceph-mon[128925]: osdmap e35: 6 total, 5 up, 6 in 2023-07-20T02:48:19.638 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:19 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T02:48:19.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:19 smithi158 ceph-mon[128925]: from='osd.5 [v2:172.21.15.188:6808/3142012805,v1:172.21.15.188:6809/3142012805]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-07-20T02:48:19.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:19 smithi158 ceph-mon[128925]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-07-20T02:48:20.303 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-20T02:48:20.303 INFO:teuthology.orchestra.run.smithi007.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-20T02:48:20.303 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10+0 records in 2023-07-20T02:48:20.303 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2023-07-20T02:48:20.304 INFO:teuthology.orchestra.run.smithi007.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0384555 s, 273 MB/s 2023-07-20T02:48:20.304 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping successful for: 2023-07-20T02:48:20.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:20 smithi188 ceph-mon[129553]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2023-07-20T02:48:20.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:20 smithi188 ceph-mon[129553]: osdmap e36: 6 total, 5 up, 6 in 2023-07-20T02:48:20.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:20 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T02:48:20.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:20 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T02:48:20.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:20 smithi188 ceph-mon[129553]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:48:20.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:20 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:20.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:20 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:20.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:20 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:48:20.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:20 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:48:20.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:20 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:20.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:20 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:48:20.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:20 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:48:20.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:20 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:20.638 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:20 smithi158 ceph-mon[128925]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2023-07-20T02:48:20.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:20 smithi158 ceph-mon[128925]: osdmap e36: 6 total, 5 up, 6 in 2023-07-20T02:48:20.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:20 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T02:48:20.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:20 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T02:48:20.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:20 smithi158 ceph-mon[128925]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:48:20.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:20 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:20.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:20 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:20.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:20 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:48:20.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:20 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:48:20.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:20 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:20.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:20 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:48:20.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:20 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:48:20.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:20 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:20.690 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:20 smithi007 ceph-mon[126353]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2023-07-20T02:48:20.690 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:20 smithi007 ceph-mon[126353]: osdmap e36: 6 total, 5 up, 6 in 2023-07-20T02:48:20.690 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:20 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T02:48:20.690 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:20 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T02:48:20.691 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:20 smithi007 ceph-mon[126353]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-20T02:48:20.691 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:20 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:20.691 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:20 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:20.691 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:20 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:48:20.691 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:20 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:48:20.691 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:20 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:20.691 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:20 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:48:20.692 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:20 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:48:20.692 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:20 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:20.701 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi007:vg_nvme/lv_2 2023-07-20T02:48:21.433 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:48:21.638 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:21 smithi158 ceph-mon[128925]: purged_snaps scrub starts 2023-07-20T02:48:21.638 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:21 smithi158 ceph-mon[128925]: purged_snaps scrub ok 2023-07-20T02:48:21.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:21 smithi158 ceph-mon[128925]: Detected new or changed devices on smithi188 2023-07-20T02:48:21.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:21 smithi158 ceph-mon[128925]: Adjusting osd_memory_target on smithi188 to 8584M 2023-07-20T02:48:21.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:21 smithi158 ceph-mon[128925]: osd.5 [v2:172.21.15.188:6808/3142012805,v1:172.21.15.188:6809/3142012805] boot 2023-07-20T02:48:21.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:21 smithi158 ceph-mon[128925]: osdmap e37: 6 total, 6 up, 6 in 2023-07-20T02:48:21.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:21 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T02:48:21.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:21 smithi158 ceph-mon[128925]: osdmap e38: 6 total, 6 up, 6 in 2023-07-20T02:48:21.658 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:21 smithi007 ceph-mon[126353]: purged_snaps scrub starts 2023-07-20T02:48:21.658 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:21 smithi007 ceph-mon[126353]: purged_snaps scrub ok 2023-07-20T02:48:21.659 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:21 smithi007 ceph-mon[126353]: Detected new or changed devices on smithi188 2023-07-20T02:48:21.659 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:21 smithi007 ceph-mon[126353]: Adjusting osd_memory_target on smithi188 to 8584M 2023-07-20T02:48:21.659 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:21 smithi007 ceph-mon[126353]: osd.5 [v2:172.21.15.188:6808/3142012805,v1:172.21.15.188:6809/3142012805] boot 2023-07-20T02:48:21.659 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:21 smithi007 ceph-mon[126353]: osdmap e37: 6 total, 6 up, 6 in 2023-07-20T02:48:21.660 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:21 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T02:48:21.660 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:21 smithi007 ceph-mon[126353]: osdmap e38: 6 total, 6 up, 6 in 2023-07-20T02:48:21.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:21 smithi188 ceph-mon[129553]: purged_snaps scrub starts 2023-07-20T02:48:21.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:21 smithi188 ceph-mon[129553]: purged_snaps scrub ok 2023-07-20T02:48:21.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:21 smithi188 ceph-mon[129553]: Detected new or changed devices on smithi188 2023-07-20T02:48:21.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:21 smithi188 ceph-mon[129553]: Adjusting osd_memory_target on smithi188 to 8584M 2023-07-20T02:48:21.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:21 smithi188 ceph-mon[129553]: osd.5 [v2:172.21.15.188:6808/3142012805,v1:172.21.15.188:6809/3142012805] boot 2023-07-20T02:48:21.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:21 smithi188 ceph-mon[129553]: osdmap e37: 6 total, 6 up, 6 in 2023-07-20T02:48:21.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:21 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T02:48:21.714 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:21 smithi188 ceph-mon[129553]: osdmap e38: 6 total, 6 up, 6 in 2023-07-20T02:48:22.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:22 smithi007 ceph-mon[126353]: pgmap v132: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:48:22.638 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:22 smithi158 ceph-mon[128925]: pgmap v132: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:48:22.711 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:22 smithi188 ceph-mon[129553]: pgmap v132: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:48:22.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.939+0000 7fa902abb700 1 -- 172.21.15.7:0/3508051785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8fc1023a0 msgr2=0x7fa8fc108c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:48:22.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.939+0000 7fa902abb700 1 --2- 172.21.15.7:0/3508051785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8fc1023a0 0x7fa8fc108c50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa8f400b0d0 tx=0x7fa8f400b3e0 comp rx=0 tx=0).stop 2023-07-20T02:48:22.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.939+0000 7fa902abb700 1 -- 172.21.15.7:0/3508051785 shutdown_connections 2023-07-20T02:48:22.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.939+0000 7fa902abb700 1 --2- 172.21.15.7:0/3508051785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8fc1023a0 0x7fa8fc108c50 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:48:22.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.939+0000 7fa902abb700 1 --2- 172.21.15.7:0/3508051785 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8fc100800 0x7fa8fc100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:48:22.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.939+0000 7fa902abb700 1 --2- 172.21.15.7:0/3508051785 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8fc101a00 0x7fa8fc101e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:48:22.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.939+0000 7fa902abb700 1 -- 172.21.15.7:0/3508051785 >> 172.21.15.7:0/3508051785 conn(0x7fa8fc0fbd60 msgr2=0x7fa8fc0fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:48:22.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.939+0000 7fa902abb700 1 -- 172.21.15.7:0/3508051785 shutdown_connections 2023-07-20T02:48:22.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.939+0000 7fa902abb700 1 -- 172.21.15.7:0/3508051785 wait complete. 2023-07-20T02:48:22.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.940+0000 7fa902abb700 1 Processor -- start 2023-07-20T02:48:22.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.941+0000 7fa902abb700 1 -- start start 2023-07-20T02:48:22.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.941+0000 7fa902abb700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8fc100800 0x7fa8fc196630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:48:22.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.941+0000 7fa902abb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8fc101a00 0x7fa8fc196b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:48:22.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.941+0000 7fa902abb700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8fc1023a0 0x7fa8fc19dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:48:22.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.941+0000 7fa902abb700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8fc19e1c0 con 0x7fa8fc101a00 2023-07-20T02:48:22.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.941+0000 7fa902abb700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8fc19e300 con 0x7fa8fc1023a0 2023-07-20T02:48:22.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.941+0000 7fa902abb700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8fc19e490 con 0x7fa8fc100800 2023-07-20T02:48:22.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.941+0000 7fa8fbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8fc101a00 0x7fa8fc196b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:48:22.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.941+0000 7fa900857700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8fc100800 0x7fa8fc196630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:48:22.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.941+0000 7fa8fbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8fc101a00 0x7fa8fc196b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:32978/0 (socket says 172.21.15.7:32978) 2023-07-20T02:48:22.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.941+0000 7fa8fbfff700 1 -- 172.21.15.7:0/508833052 learned_addr learned my addr 172.21.15.7:0/508833052 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:48:22.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.941+0000 7fa901058700 1 --2- 172.21.15.7:0/508833052 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8fc1023a0 0x7fa8fc19dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:48:22.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.942+0000 7fa8fbfff700 1 -- 172.21.15.7:0/508833052 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8fc100800 msgr2=0x7fa8fc196630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:48:22.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.942+0000 7fa8fbfff700 1 --2- 172.21.15.7:0/508833052 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8fc100800 0x7fa8fc196630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:48:22.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.942+0000 7fa8fbfff700 1 -- 172.21.15.7:0/508833052 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8fc1023a0 msgr2=0x7fa8fc19dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:48:22.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.942+0000 7fa8fbfff700 1 --2- 172.21.15.7:0/508833052 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8fc1023a0 0x7fa8fc19dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:48:22.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.942+0000 7fa8fbfff700 1 -- 172.21.15.7:0/508833052 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f4009d00 con 0x7fa8fc101a00 2023-07-20T02:48:22.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.942+0000 7fa901058700 1 --2- 172.21.15.7:0/508833052 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8fc1023a0 0x7fa8fc19dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T02:48:22.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.942+0000 7fa900857700 1 --2- 172.21.15.7:0/508833052 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8fc100800 0x7fa8fc196630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T02:48:22.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.942+0000 7fa8fbfff700 1 --2- 172.21.15.7:0/508833052 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8fc101a00 0x7fa8fc196b70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa8e800d9b0 tx=0x7fa8e800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:48:22.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.942+0000 7fa8f9ffb700 1 -- 172.21.15.7:0/508833052 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e8004a30 con 0x7fa8fc101a00 2023-07-20T02:48:22.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.942+0000 7fa902abb700 1 -- 172.21.15.7:0/508833052 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8fc19e770 con 0x7fa8fc101a00 2023-07-20T02:48:22.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.942+0000 7fa8f9ffb700 1 -- 172.21.15.7:0/508833052 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa8e80054e0 con 0x7fa8fc101a00 2023-07-20T02:48:22.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.942+0000 7fa8f9ffb700 1 -- 172.21.15.7:0/508833052 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e8009b20 con 0x7fa8fc101a00 2023-07-20T02:48:22.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.943+0000 7fa902abb700 1 -- 172.21.15.7:0/508833052 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8fc19ec70 con 0x7fa8fc101a00 2023-07-20T02:48:22.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.944+0000 7fa902abb700 1 -- 172.21.15.7:0/508833052 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8fc107e00 con 0x7fa8fc101a00 2023-07-20T02:48:22.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.946+0000 7fa8f9ffb700 1 -- 172.21.15.7:0/508833052 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fa8e8009c80 con 0x7fa8fc101a00 2023-07-20T02:48:22.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.947+0000 7fa8f9ffb700 1 --2- 172.21.15.7:0/508833052 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fa8ec0a11d0 0x7fa8ec0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:48:22.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.947+0000 7fa8f9ffb700 1 -- 172.21.15.7:0/508833052 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4562+0+0 (secure 0 0 0) 0x7fa8e80cb820 con 0x7fa8fc101a00 2023-07-20T02:48:22.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.947+0000 7fa900857700 1 --2- 172.21.15.7:0/508833052 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fa8ec0a11d0 0x7fa8ec0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:48:22.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.948+0000 7fa900857700 1 --2- 172.21.15.7:0/508833052 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fa8ec0a11d0 0x7fa8ec0a3690 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc101860 tx=0x7fa8f0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:48:22.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:22.951+0000 7fa8f9ffb700 1 -- 172.21.15.7:0/508833052 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa8e8095d70 con 0x7fa8fc101a00 2023-07-20T02:48:23.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:23.168+0000 7fa902abb700 1 -- 172.21.15.7:0/508833052 --> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8fc060bb0 con 0x7fa8ec0a11d0 2023-07-20T02:48:23.638 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:23 smithi158 ceph-mon[128925]: osdmap e39: 6 total, 6 up, 6 in 2023-07-20T02:48:23.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:23 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:48:23.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:23 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:48:23.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:23 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:48:23.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:23 smithi007 ceph-mon[126353]: osdmap e39: 6 total, 6 up, 6 in 2023-07-20T02:48:23.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:23 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:48:23.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:23 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:48:23.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:23 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:48:23.711 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:23 smithi188 ceph-mon[129553]: osdmap e39: 6 total, 6 up, 6 in 2023-07-20T02:48:23.711 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:23 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:48:23.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:23 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:48:23.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:23 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:48:24.514 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:24 smithi007 ceph-mon[126353]: from='client.14331 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:48:24.514 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:24 smithi007 ceph-mon[126353]: osdmap e40: 6 total, 6 up, 6 in 2023-07-20T02:48:24.514 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:24 smithi007 ceph-mon[126353]: pgmap v135: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:48:24.638 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:24 smithi158 ceph-mon[128925]: from='client.14331 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:48:24.638 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:24 smithi158 ceph-mon[128925]: osdmap e40: 6 total, 6 up, 6 in 2023-07-20T02:48:24.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:24 smithi158 ceph-mon[128925]: pgmap v135: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:48:24.711 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:24 smithi188 ceph-mon[129553]: from='client.14331 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:48:24.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:24 smithi188 ceph-mon[129553]: osdmap e40: 6 total, 6 up, 6 in 2023-07-20T02:48:24.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:24 smithi188 ceph-mon[129553]: pgmap v135: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:48:25.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:25 smithi007 ceph-mon[126353]: osdmap e41: 6 total, 6 up, 6 in 2023-07-20T02:48:25.638 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:25 smithi158 ceph-mon[128925]: osdmap e41: 6 total, 6 up, 6 in 2023-07-20T02:48:25.711 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:25 smithi188 ceph-mon[129553]: osdmap e41: 6 total, 6 up, 6 in 2023-07-20T02:48:26.638 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:26 smithi158 ceph-mon[128925]: pgmap v137: 1 pgs: 1 remapped+peering; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:48:26.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:26 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/2754124590' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82ef2d33-1045-41f1-bf69-b03404bcace8"}]: dispatch 2023-07-20T02:48:26.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:26 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/2754124590' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "82ef2d33-1045-41f1-bf69-b03404bcace8"}]': finished 2023-07-20T02:48:26.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:26 smithi158 ceph-mon[128925]: osdmap e42: 7 total, 6 up, 7 in 2023-07-20T02:48:26.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:26 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T02:48:26.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:26 smithi007 ceph-mon[126353]: pgmap v137: 1 pgs: 1 remapped+peering; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:48:26.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:26 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/2754124590' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82ef2d33-1045-41f1-bf69-b03404bcace8"}]: dispatch 2023-07-20T02:48:26.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:26 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/2754124590' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "82ef2d33-1045-41f1-bf69-b03404bcace8"}]': finished 2023-07-20T02:48:26.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:26 smithi007 ceph-mon[126353]: osdmap e42: 7 total, 6 up, 7 in 2023-07-20T02:48:26.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:26 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T02:48:26.711 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:26 smithi188 ceph-mon[129553]: pgmap v137: 1 pgs: 1 remapped+peering; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:48:26.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:26 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/2754124590' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82ef2d33-1045-41f1-bf69-b03404bcace8"}]: dispatch 2023-07-20T02:48:26.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:26 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/2754124590' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "82ef2d33-1045-41f1-bf69-b03404bcace8"}]': finished 2023-07-20T02:48:26.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:26 smithi188 ceph-mon[129553]: osdmap e42: 7 total, 6 up, 7 in 2023-07-20T02:48:26.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:26 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T02:48:27.638 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:27 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/3660856147' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:48:27.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:27 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/3660856147' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:48:27.711 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:27 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/3660856147' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:48:28.638 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:28 smithi158 ceph-mon[128925]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 110 KiB/s, 0 objects/s recovering 2023-07-20T02:48:28.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:28 smithi007 ceph-mon[126353]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 110 KiB/s, 0 objects/s recovering 2023-07-20T02:48:28.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:28 smithi188 ceph-mon[129553]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 110 KiB/s, 0 objects/s recovering 2023-07-20T02:48:30.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:30 smithi158 ceph-mon[128925]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 93 KiB/s, 0 objects/s recovering 2023-07-20T02:48:30.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:30 smithi007 ceph-mon[126353]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 93 KiB/s, 0 objects/s recovering 2023-07-20T02:48:30.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:30 smithi188 ceph-mon[129553]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 93 KiB/s, 0 objects/s recovering 2023-07-20T02:48:32.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:32 smithi007 ceph-mon[126353]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T02:48:32.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:32 smithi158 ceph-mon[128925]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T02:48:32.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:32 smithi188 ceph-mon[129553]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T02:48:34.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:34 smithi158 ceph-mon[128925]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-07-20T02:48:34.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:34 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-20T02:48:34.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:34 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:48:34.912 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:34 smithi007 ceph-mon[126353]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-07-20T02:48:34.913 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:34 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-20T02:48:34.913 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:34 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:48:34.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:34 smithi188 ceph-mon[129553]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-07-20T02:48:34.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:34 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-20T02:48:34.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:34 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:48:35.855 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:35 smithi007 ceph-mon[126353]: Deploying daemon osd.6 on smithi007 2023-07-20T02:48:35.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:35 smithi158 ceph-mon[128925]: Deploying daemon osd.6 on smithi007 2023-07-20T02:48:35.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:35 smithi188 ceph-mon[129553]: Deploying daemon osd.6 on smithi007 2023-07-20T02:48:36.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:36 smithi007 ceph-mon[126353]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-20T02:48:36.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:36 smithi158 ceph-mon[128925]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-20T02:48:36.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:36 smithi188 ceph-mon[129553]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-20T02:48:38.388 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:38 smithi158 ceph-mon[128925]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:48:38.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:38 smithi007 ceph-mon[126353]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:48:38.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:38 smithi188 ceph-mon[129553]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:48:40.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:40 smithi158 ceph-mon[128925]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:48:40.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:40 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:40.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:40 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:48:40.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:40 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:40.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:40 smithi007 ceph-mon[126353]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:48:40.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:40 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:40.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:40 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:48:40.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:40 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:40.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:40 smithi188 ceph-mon[129553]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:48:40.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:40 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:40.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:40 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:48:40.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:40 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:42.888 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:42 smithi158 ceph-mon[128925]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:48:42.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:42 smithi007 ceph-mon[126353]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:48:42.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:42 smithi188 ceph-mon[129553]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:48:43.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:43.980+0000 7fa8f9ffb700 1 -- 172.21.15.7:0/508833052 <== mgr.14152 v2:172.21.15.7:6800/1354542137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa8fc060bb0 con 0x7fa8ec0a11d0 2023-07-20T02:48:43.993 INFO:teuthology.orchestra.run.smithi007.stdout:Created osd(s) 6 on host 'smithi007' 2023-07-20T02:48:43.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:43.983+0000 7fa902abb700 1 -- 172.21.15.7:0/508833052 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fa8ec0a11d0 msgr2=0x7fa8ec0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:48:43.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:43.983+0000 7fa902abb700 1 --2- 172.21.15.7:0/508833052 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fa8ec0a11d0 0x7fa8ec0a3690 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc101860 tx=0x7fa8f0009260 comp rx=0 tx=0).stop 2023-07-20T02:48:43.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:43.983+0000 7fa902abb700 1 -- 172.21.15.7:0/508833052 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8fc101a00 msgr2=0x7fa8fc196b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:48:43.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:43.983+0000 7fa902abb700 1 --2- 172.21.15.7:0/508833052 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8fc101a00 0x7fa8fc196b70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa8e800d9b0 tx=0x7fa8e800dcc0 comp rx=0 tx=0).stop 2023-07-20T02:48:43.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:43.983+0000 7fa902abb700 1 -- 172.21.15.7:0/508833052 shutdown_connections 2023-07-20T02:48:43.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:43.983+0000 7fa902abb700 1 --2- 172.21.15.7:0/508833052 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa8fc1023a0 0x7fa8fc19dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:48:43.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:43.983+0000 7fa902abb700 1 --2- 172.21.15.7:0/508833052 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8fc100800 0x7fa8fc196630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:48:43.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:43.983+0000 7fa902abb700 1 --2- 172.21.15.7:0/508833052 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fa8ec0a11d0 0x7fa8ec0a3690 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:48:43.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:43.983+0000 7fa902abb700 1 --2- 172.21.15.7:0/508833052 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8fc101a00 0x7fa8fc196b70 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:48:43.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:43.983+0000 7fa902abb700 1 -- 172.21.15.7:0/508833052 >> 172.21.15.7:0/508833052 conn(0x7fa8fc0fbd60 msgr2=0x7fa8fc104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:48:43.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:43.983+0000 7fa902abb700 1 -- 172.21.15.7:0/508833052 shutdown_connections 2023-07-20T02:48:43.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:48:43.983+0000 7fa902abb700 1 -- 172.21.15.7:0/508833052 wait complete. 2023-07-20T02:48:44.364 DEBUG:teuthology.orchestra.run.smithi007:osd.6> sudo journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.6.service 2023-07-20T02:48:44.370 INFO:tasks.cephadm:Deploying osd.7 on smithi158 with /dev/vg_nvme/lv_2... 2023-07-20T02:48:44.371 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-20T02:48:44.444 INFO:journalctl@ceph.osd.6.smithi007.stdout:-- Logs begin at Thu 2023-07-20 02:22:32 UTC. -- 2023-07-20T02:48:44.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:44 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:44.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:44 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:44.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:44 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:48:44.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:44 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:48:44.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:44 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:44.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:44 smithi007 ceph-mon[126353]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:48:44.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:44 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:44.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:44 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:44.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:44 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:48:44.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:44 smithi007 ceph-mon[126353]: from='osd.6 [v2:172.21.15.7:6818/859560576,v1:172.21.15.7:6819/859560576]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-20T02:48:44.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:44 smithi007 ceph-mon[126353]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-20T02:48:44.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:44 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:44.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:44 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:44.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:44 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:48:44.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:44 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:48:44.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:44 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:44.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:44 smithi188 ceph-mon[129553]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:48:44.714 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:44 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:44.714 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:44 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:44.714 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:44 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:48:44.714 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:44 smithi188 ceph-mon[129553]: from='osd.6 [v2:172.21.15.7:6818/859560576,v1:172.21.15.7:6819/859560576]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-20T02:48:44.714 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:44 smithi188 ceph-mon[129553]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-20T02:48:44.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:44 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:44.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:44 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:44.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:44 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:48:44.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:44 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:48:44.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:44 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:44.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:44 smithi158 ceph-mon[128925]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:48:44.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:44 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:44.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:44 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:44.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:44 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:48:44.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:44 smithi158 ceph-mon[128925]: from='osd.6 [v2:172.21.15.7:6818/859560576,v1:172.21.15.7:6819/859560576]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-20T02:48:44.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:44 smithi158 ceph-mon[128925]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-20T02:48:44.933 INFO:teuthology.orchestra.run.smithi158.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T02:48:46.154 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:45 smithi007 ceph-mon[126353]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-20T02:48:46.155 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:45 smithi007 ceph-mon[126353]: osdmap e43: 7 total, 6 up, 7 in 2023-07-20T02:48:46.155 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:45 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T02:48:46.155 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:45 smithi007 ceph-mon[126353]: from='osd.6 [v2:172.21.15.7:6818/859560576,v1:172.21.15.7:6819/859560576]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-07-20T02:48:46.155 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:45 smithi007 ceph-mon[126353]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-07-20T02:48:46.155 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:45 smithi007 ceph-mon[126353]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:48:46.156 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:48:45 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:48:45.951+0000 7f7814176700 -1 osd.6 0 waiting for initial osdmap 2023-07-20T02:48:46.156 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:48:45 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:48:45.962+0000 7f780ff6c700 -1 osd.6 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T02:48:46.211 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:45 smithi188 ceph-mon[129553]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-20T02:48:46.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:45 smithi188 ceph-mon[129553]: osdmap e43: 7 total, 6 up, 7 in 2023-07-20T02:48:46.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:45 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T02:48:46.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:45 smithi188 ceph-mon[129553]: from='osd.6 [v2:172.21.15.7:6818/859560576,v1:172.21.15.7:6819/859560576]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-07-20T02:48:46.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:45 smithi188 ceph-mon[129553]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-07-20T02:48:46.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:45 smithi188 ceph-mon[129553]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:48:46.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:45 smithi158 ceph-mon[128925]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-20T02:48:46.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:45 smithi158 ceph-mon[128925]: osdmap e43: 7 total, 6 up, 7 in 2023-07-20T02:48:46.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:45 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T02:48:46.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:45 smithi158 ceph-mon[128925]: from='osd.6 [v2:172.21.15.7:6818/859560576,v1:172.21.15.7:6819/859560576]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-07-20T02:48:46.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:45 smithi158 ceph-mon[128925]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-07-20T02:48:46.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:45 smithi158 ceph-mon[128925]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-20T02:48:46.401 INFO:teuthology.orchestra.run.smithi158.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-20T02:48:46.401 INFO:teuthology.orchestra.run.smithi158.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-20T02:48:46.401 INFO:teuthology.orchestra.run.smithi158.stderr: stderr: 10+0 records in 2023-07-20T02:48:46.401 INFO:teuthology.orchestra.run.smithi158.stderr:10+0 records out 2023-07-20T02:48:46.402 INFO:teuthology.orchestra.run.smithi158.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00826293 s, 1.3 GB/s 2023-07-20T02:48:46.402 INFO:teuthology.orchestra.run.smithi158.stderr:--> Zapping successful for: 2023-07-20T02:48:46.789 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi158:vg_nvme/lv_2 2023-07-20T02:48:47.138 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:46 smithi158 ceph-mon[128925]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2023-07-20T02:48:47.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:46 smithi158 ceph-mon[128925]: osdmap e44: 7 total, 6 up, 7 in 2023-07-20T02:48:47.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:46 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T02:48:47.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:46 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T02:48:47.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:46 smithi007 ceph-mon[126353]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2023-07-20T02:48:47.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:46 smithi007 ceph-mon[126353]: osdmap e44: 7 total, 6 up, 7 in 2023-07-20T02:48:47.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:46 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T02:48:47.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:46 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T02:48:47.343 INFO:teuthology.orchestra.run.smithi158.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T02:48:47.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:46 smithi188 ceph-mon[129553]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2023-07-20T02:48:47.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:46 smithi188 ceph-mon[129553]: osdmap e44: 7 total, 6 up, 7 in 2023-07-20T02:48:47.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:46 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T02:48:47.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:46 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T02:48:48.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:47 smithi158 ceph-mon[128925]: purged_snaps scrub starts 2023-07-20T02:48:48.140 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:47 smithi158 ceph-mon[128925]: purged_snaps scrub ok 2023-07-20T02:48:48.140 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:47 smithi158 ceph-mon[128925]: osd.6 [v2:172.21.15.7:6818/859560576,v1:172.21.15.7:6819/859560576] boot 2023-07-20T02:48:48.140 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:47 smithi158 ceph-mon[128925]: osdmap e45: 7 total, 7 up, 7 in 2023-07-20T02:48:48.140 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:47 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T02:48:48.140 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:47 smithi158 ceph-mon[128925]: Detected new or changed devices on smithi007 2023-07-20T02:48:48.140 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:47 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:48.140 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:47 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:48.141 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:47 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:48:48.141 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:47 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:48:48.141 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:47 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:48:48.141 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:47 smithi158 ceph-mon[128925]: Adjusting osd_memory_target on smithi007 to 5757M 2023-07-20T02:48:48.141 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:47 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:48.141 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:47 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:48:48.141 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:47 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:48:48.142 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:47 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:48.142 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:47 smithi158 ceph-mon[128925]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:48:48.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:47 smithi007 ceph-mon[126353]: purged_snaps scrub starts 2023-07-20T02:48:48.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:47 smithi007 ceph-mon[126353]: purged_snaps scrub ok 2023-07-20T02:48:48.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:47 smithi007 ceph-mon[126353]: osd.6 [v2:172.21.15.7:6818/859560576,v1:172.21.15.7:6819/859560576] boot 2023-07-20T02:48:48.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:47 smithi007 ceph-mon[126353]: osdmap e45: 7 total, 7 up, 7 in 2023-07-20T02:48:48.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:47 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T02:48:48.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:47 smithi007 ceph-mon[126353]: Detected new or changed devices on smithi007 2023-07-20T02:48:48.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:47 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:48.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:47 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:48.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:47 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:48:48.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:47 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:48:48.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:47 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:48:48.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:47 smithi007 ceph-mon[126353]: Adjusting osd_memory_target on smithi007 to 5757M 2023-07-20T02:48:48.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:47 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:48.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:47 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:48:48.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:47 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:48:48.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:47 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:48.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:47 smithi007 ceph-mon[126353]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:48:48.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.437+0000 7f6547da1700 1 -- 172.21.15.158:0/40191463 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65400ff730 msgr2=0x7f65400ffb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:48:48.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.437+0000 7f6547da1700 1 --2- 172.21.15.158:0/40191463 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65400ff730 0x7f65400ffb50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f65300099c0 tx=0x7f6530009cd0 comp rx=0 tx=0).stop 2023-07-20T02:48:48.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.437+0000 7f6547da1700 1 -- 172.21.15.158:0/40191463 shutdown_connections 2023-07-20T02:48:48.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.437+0000 7f6547da1700 1 --2- 172.21.15.158:0/40191463 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65401012d0 0x7f6540107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:48:48.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.437+0000 7f6547da1700 1 --2- 172.21.15.158:0/40191463 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65400ff730 0x7f65400ffb50 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:48:48.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.437+0000 7f6547da1700 1 --2- 172.21.15.158:0/40191463 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6540100930 0x7f6540100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:48:48.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.437+0000 7f6547da1700 1 -- 172.21.15.158:0/40191463 >> 172.21.15.158:0/40191463 conn(0x7f65400facf0 msgr2=0x7f65400fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:48:48.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.437+0000 7f6547da1700 1 -- 172.21.15.158:0/40191463 shutdown_connections 2023-07-20T02:48:48.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.437+0000 7f6547da1700 1 -- 172.21.15.158:0/40191463 wait complete. 2023-07-20T02:48:48.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.438+0000 7f6547da1700 1 Processor -- start 2023-07-20T02:48:48.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.438+0000 7f6547da1700 1 -- start start 2023-07-20T02:48:48.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.438+0000 7f6547da1700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65400ff730 0x7f6540196560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:48:48.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.438+0000 7f6547da1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6540100930 0x7f6540196aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:48:48.452 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.438+0000 7f6547da1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65401012d0 0x7f654019db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:48:48.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.438+0000 7f6547da1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f654019e0f0 con 0x7f6540100930 2023-07-20T02:48:48.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.438+0000 7f6547da1700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f654019e230 con 0x7f65400ff730 2023-07-20T02:48:48.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.438+0000 7f6547da1700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f654019e3c0 con 0x7f65401012d0 2023-07-20T02:48:48.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.438+0000 7f6545b3d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65400ff730 0x7f6540196560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:48:48.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.438+0000 7f6545b3d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65400ff730 0x7f6540196560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58374/0 (socket says 172.21.15.158:58374) 2023-07-20T02:48:48.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.438+0000 7f6545b3d700 1 -- 172.21.15.158:0/1275604081 learned_addr learned my addr 172.21.15.158:0/1275604081 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-07-20T02:48:48.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.438+0000 7f654533c700 1 --2- 172.21.15.158:0/1275604081 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6540100930 0x7f6540196aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:48:48.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.438+0000 7f654633e700 1 --2- 172.21.15.158:0/1275604081 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65401012d0 0x7f654019db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:48:48.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.438+0000 7f6545b3d700 1 -- 172.21.15.158:0/1275604081 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65401012d0 msgr2=0x7f654019db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:48:48.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.438+0000 7f6545b3d700 1 --2- 172.21.15.158:0/1275604081 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65401012d0 0x7f654019db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:48:48.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.438+0000 7f6545b3d700 1 -- 172.21.15.158:0/1275604081 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6540100930 msgr2=0x7f6540196aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:48:48.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.438+0000 7f6545b3d700 1 --2- 172.21.15.158:0/1275604081 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6540100930 0x7f6540196aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:48:48.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.438+0000 7f6545b3d700 1 -- 172.21.15.158:0/1275604081 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65300095f0 con 0x7f65400ff730 2023-07-20T02:48:48.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.438+0000 7f654633e700 1 --2- 172.21.15.158:0/1275604081 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65401012d0 0x7f654019db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T02:48:48.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.439+0000 7f6545b3d700 1 --2- 172.21.15.158:0/1275604081 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65400ff730 0x7f6540196560 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6530009990 tx=0x7f6530003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:48:48.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.439+0000 7f6536ffd700 1 -- 172.21.15.158:0/1275604081 <== mon.2 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653001d070 con 0x7f65400ff730 2023-07-20T02:48:48.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.439+0000 7f6536ffd700 1 -- 172.21.15.158:0/1275604081 <== mon.2 v2:172.21.15.158:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6530004640 con 0x7f65400ff730 2023-07-20T02:48:48.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.439+0000 7f6547da1700 1 -- 172.21.15.158:0/1275604081 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f654019e640 con 0x7f65400ff730 2023-07-20T02:48:48.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.439+0000 7f6536ffd700 1 -- 172.21.15.158:0/1275604081 <== mon.2 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653000f460 con 0x7f65400ff730 2023-07-20T02:48:48.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.439+0000 7f6547da1700 1 -- 172.21.15.158:0/1275604081 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f654019eae0 con 0x7f65400ff730 2023-07-20T02:48:48.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.440+0000 7f6547da1700 1 -- 172.21.15.158:0/1275604081 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6540107870 con 0x7f65400ff730 2023-07-20T02:48:48.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.441+0000 7f6536ffd700 1 -- 172.21.15.158:0/1275604081 <== mon.2 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f6530004050 con 0x7f65400ff730 2023-07-20T02:48:48.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.441+0000 7f6536ffd700 1 --2- 172.21.15.158:0/1275604081 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f652c0a1150 0x7f652c0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:48:48.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.441+0000 7f6536ffd700 1 -- 172.21.15.158:0/1275604081 <== mon.2 v2:172.21.15.158:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f65300cd110 con 0x7f65400ff730 2023-07-20T02:48:48.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.442+0000 7f654533c700 1 --2- 172.21.15.158:0/1275604081 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f652c0a1150 0x7f652c0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:48:48.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.442+0000 7f654533c700 1 --2- 172.21.15.158:0/1275604081 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f652c0a1150 0x7f652c0a3610 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f653c005fd0 tx=0x7f653c00a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:48:48.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.446+0000 7f6536ffd700 1 -- 172.21.15.158:0/1275604081 <== mon.2 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6530097600 con 0x7f65400ff730 2023-07-20T02:48:48.461 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:47 smithi188 ceph-mon[129553]: purged_snaps scrub starts 2023-07-20T02:48:48.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:47 smithi188 ceph-mon[129553]: purged_snaps scrub ok 2023-07-20T02:48:48.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:47 smithi188 ceph-mon[129553]: osd.6 [v2:172.21.15.7:6818/859560576,v1:172.21.15.7:6819/859560576] boot 2023-07-20T02:48:48.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:47 smithi188 ceph-mon[129553]: osdmap e45: 7 total, 7 up, 7 in 2023-07-20T02:48:48.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:47 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T02:48:48.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:47 smithi188 ceph-mon[129553]: Detected new or changed devices on smithi007 2023-07-20T02:48:48.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:47 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:48.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:47 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:48.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:47 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:48:48.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:47 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:48:48.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:47 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:48:48.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:47 smithi188 ceph-mon[129553]: Adjusting osd_memory_target on smithi007 to 5757M 2023-07-20T02:48:48.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:47 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:48.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:47 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:48:48.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:47 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:48:48.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:47 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:48:48.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:47 smithi188 ceph-mon[129553]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:48:48.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:48:48.638+0000 7f6547da1700 1 -- 172.21.15.158:0/1275604081 --> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi158:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f6540060bb0 con 0x7f652c0a1150 2023-07-20T02:48:49.625 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:49 smithi158 ceph-mon[128925]: osdmap e46: 7 total, 7 up, 7 in 2023-07-20T02:48:49.625 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:49 smithi158 ceph-mon[128925]: from='client.24299 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi158:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:48:49.625 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:49 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:48:49.625 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:49 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:48:49.625 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:49 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:48:49.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:49 smithi007 ceph-mon[126353]: osdmap e46: 7 total, 7 up, 7 in 2023-07-20T02:48:49.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:49 smithi007 ceph-mon[126353]: from='client.24299 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi158:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:48:49.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:49 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:48:49.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:49 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:48:49.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:49 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:48:49.711 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:49 smithi188 ceph-mon[129553]: osdmap e46: 7 total, 7 up, 7 in 2023-07-20T02:48:49.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:49 smithi188 ceph-mon[129553]: from='client.24299 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi158:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:48:49.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:49 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:48:49.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:49 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:48:49.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:49 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:48:50.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:50 smithi158 ceph-mon[128925]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:48:50.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:50 smithi007 ceph-mon[126353]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:48:50.711 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:50 smithi188 ceph-mon[129553]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:48:51.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:51 smithi158 ceph-mon[128925]: from='client.? 172.21.15.158:0/295440229' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "020ec859-3fc2-4da2-a0dc-10530a4a366c"}]: dispatch 2023-07-20T02:48:51.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:51 smithi158 ceph-mon[128925]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "020ec859-3fc2-4da2-a0dc-10530a4a366c"}]: dispatch 2023-07-20T02:48:51.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:51 smithi158 ceph-mon[128925]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "020ec859-3fc2-4da2-a0dc-10530a4a366c"}]': finished 2023-07-20T02:48:51.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:51 smithi158 ceph-mon[128925]: osdmap e47: 8 total, 7 up, 8 in 2023-07-20T02:48:51.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:51 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:48:51.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:51 smithi007 ceph-mon[126353]: from='client.? 172.21.15.158:0/295440229' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "020ec859-3fc2-4da2-a0dc-10530a4a366c"}]: dispatch 2023-07-20T02:48:51.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:51 smithi007 ceph-mon[126353]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "020ec859-3fc2-4da2-a0dc-10530a4a366c"}]: dispatch 2023-07-20T02:48:51.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:51 smithi007 ceph-mon[126353]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "020ec859-3fc2-4da2-a0dc-10530a4a366c"}]': finished 2023-07-20T02:48:51.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:51 smithi007 ceph-mon[126353]: osdmap e47: 8 total, 7 up, 8 in 2023-07-20T02:48:51.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:51 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:48:51.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:51 smithi188 ceph-mon[129553]: from='client.? 172.21.15.158:0/295440229' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "020ec859-3fc2-4da2-a0dc-10530a4a366c"}]: dispatch 2023-07-20T02:48:51.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:51 smithi188 ceph-mon[129553]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "020ec859-3fc2-4da2-a0dc-10530a4a366c"}]: dispatch 2023-07-20T02:48:51.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:51 smithi188 ceph-mon[129553]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "020ec859-3fc2-4da2-a0dc-10530a4a366c"}]': finished 2023-07-20T02:48:51.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:51 smithi188 ceph-mon[129553]: osdmap e47: 8 total, 7 up, 8 in 2023-07-20T02:48:51.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:51 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:48:52.638 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:52 smithi158 ceph-mon[128925]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:48:52.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:52 smithi158 ceph-mon[128925]: from='client.? 172.21.15.158:0/1908351970' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:48:52.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:52 smithi007 ceph-mon[126353]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:48:52.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:52 smithi007 ceph-mon[126353]: from='client.? 172.21.15.158:0/1908351970' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:48:52.711 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:52 smithi188 ceph-mon[129553]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:48:52.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:52 smithi188 ceph-mon[129553]: from='client.? 172.21.15.158:0/1908351970' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:48:54.824 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:54 smithi158 ceph-mon[128925]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:48:54.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:54 smithi007 ceph-mon[126353]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:48:54.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:54 smithi188 ceph-mon[129553]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:48:56.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:56 smithi158 ceph-mon[128925]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:48:56.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:56 smithi007 ceph-mon[126353]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:48:56.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:56 smithi188 ceph-mon[129553]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:48:58.348 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:58 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-20T02:48:58.348 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:58 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:48:58.348 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:58 smithi158 ceph-mon[128925]: Deploying daemon osd.7 on smithi158 2023-07-20T02:48:58.349 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:48:58 smithi158 ceph-mon[128925]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:48:58.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:58 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-20T02:48:58.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:58 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:48:58.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:58 smithi007 ceph-mon[126353]: Deploying daemon osd.7 on smithi158 2023-07-20T02:48:58.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:48:58 smithi007 ceph-mon[126353]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:48:58.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:58 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-20T02:48:58.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:58 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:48:58.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:58 smithi188 ceph-mon[129553]: Deploying daemon osd.7 on smithi158 2023-07-20T02:48:58.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:48:58 smithi188 ceph-mon[129553]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:49:00.840 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:00 smithi158 ceph-mon[128925]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:49:00.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:00 smithi007 ceph-mon[126353]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:49:00.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:00 smithi188 ceph-mon[129553]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:49:02.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:02 smithi158 ceph-mon[128925]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:49:02.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:02 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:49:02.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:02 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:02.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:02 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:02.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:02 smithi007 ceph-mon[126353]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:49:02.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:02 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:49:02.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:02 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:02.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:02 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:02.961 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:02 smithi188 ceph-mon[129553]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:49:02.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:02 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:49:02.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:02 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:02.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:02 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:04.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:04 smithi158 ceph-mon[128925]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:49:04.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:04 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:04.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:04 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:04.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:04 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:49:04.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:04 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:49:04.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:04 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:04.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:04.700+0000 7f6536ffd700 1 -- 172.21.15.158:0/1275604081 <== mgr.14152 v2:172.21.15.7:6800/1354542137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6540060bb0 con 0x7f652c0a1150 2023-07-20T02:49:04.714 INFO:teuthology.orchestra.run.smithi158.stdout:Created osd(s) 7 on host 'smithi158' 2023-07-20T02:49:04.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:04.703+0000 7f6547da1700 1 -- 172.21.15.158:0/1275604081 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f652c0a1150 msgr2=0x7f652c0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:49:04.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:04.703+0000 7f6547da1700 1 --2- 172.21.15.158:0/1275604081 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f652c0a1150 0x7f652c0a3610 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f653c005fd0 tx=0x7f653c00a560 comp rx=0 tx=0).stop 2023-07-20T02:49:04.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:04.703+0000 7f6547da1700 1 -- 172.21.15.158:0/1275604081 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65400ff730 msgr2=0x7f6540196560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:49:04.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:04.703+0000 7f6547da1700 1 --2- 172.21.15.158:0/1275604081 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65400ff730 0x7f6540196560 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6530009990 tx=0x7f6530003ba0 comp rx=0 tx=0).stop 2023-07-20T02:49:04.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:04.703+0000 7f6547da1700 1 -- 172.21.15.158:0/1275604081 shutdown_connections 2023-07-20T02:49:04.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:04.703+0000 7f6547da1700 1 --2- 172.21.15.158:0/1275604081 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65401012d0 0x7f654019db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:49:04.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:04.703+0000 7f6547da1700 1 --2- 172.21.15.158:0/1275604081 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f65400ff730 0x7f6540196560 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:49:04.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:04.703+0000 7f6547da1700 1 --2- 172.21.15.158:0/1275604081 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f652c0a1150 0x7f652c0a3610 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:49:04.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:04.703+0000 7f6547da1700 1 --2- 172.21.15.158:0/1275604081 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6540100930 0x7f6540196aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:49:04.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:04.703+0000 7f6547da1700 1 -- 172.21.15.158:0/1275604081 >> 172.21.15.158:0/1275604081 conn(0x7f65400facf0 msgr2=0x7f65400fcf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:49:04.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:04.703+0000 7f6547da1700 1 -- 172.21.15.158:0/1275604081 shutdown_connections 2023-07-20T02:49:04.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:04.703+0000 7f6547da1700 1 -- 172.21.15.158:0/1275604081 wait complete. 2023-07-20T02:49:04.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:04 smithi007 ceph-mon[126353]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:49:04.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:04 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:04.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:04 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:04.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:04 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:49:04.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:04 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:49:04.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:04 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:04.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:04 smithi188 ceph-mon[129553]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:49:04.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:04 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:04.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:04 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:04.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:04 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:49:04.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:04 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:49:04.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:04 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:05.127 DEBUG:teuthology.orchestra.run.smithi158:osd.7> sudo journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.7.service 2023-07-20T02:49:05.129 INFO:tasks.cephadm:Deploying osd.8 on smithi188 with /dev/vg_nvme/lv_2... 2023-07-20T02:49:05.130 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-20T02:49:05.160 INFO:journalctl@ceph.osd.7.smithi158.stdout:-- Logs begin at Thu 2023-07-20 02:10:12 UTC. -- 2023-07-20T02:49:05.693 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T02:49:05.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:05 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:05.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:05 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:05.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:05 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:49:05.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:05 smithi158 ceph-mon[128925]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:49:05.890 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:49:05 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:49:05.724+0000 7f6a28409640 -1 osd.7 0 log_to_monitors true 2023-07-20T02:49:05.935 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:05 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:05.935 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:05 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:05.936 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:05 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:49:05.936 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:05 smithi188 ceph-mon[129553]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:49:05.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:05 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:05.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:05 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:05.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:05 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:49:05.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:05 smithi007 ceph-mon[126353]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:49:06.943 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:06 smithi158 ceph-mon[128925]: from='osd.7 [v2:172.21.15.158:6816/2578758635,v1:172.21.15.158:6817/2578758635]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T02:49:06.944 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:06 smithi158 ceph-mon[128925]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T02:49:06.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:06 smithi007 ceph-mon[126353]: from='osd.7 [v2:172.21.15.158:6816/2578758635,v1:172.21.15.158:6817/2578758635]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T02:49:06.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:06 smithi007 ceph-mon[126353]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T02:49:06.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:06 smithi188 ceph-mon[129553]: from='osd.7 [v2:172.21.15.158:6816/2578758635,v1:172.21.15.158:6817/2578758635]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T02:49:06.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:06 smithi188 ceph-mon[129553]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T02:49:07.225 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-20T02:49:07.225 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-20T02:49:07.225 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2023-07-20T02:49:07.225 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2023-07-20T02:49:07.225 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110173 s, 952 MB/s 2023-07-20T02:49:07.226 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2023-07-20T02:49:07.746 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi188:vg_nvme/lv_2 2023-07-20T02:49:07.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:07 smithi007 ceph-mon[126353]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-20T02:49:07.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:07 smithi007 ceph-mon[126353]: osdmap e48: 8 total, 7 up, 8 in 2023-07-20T02:49:07.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:07 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:49:07.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:07 smithi007 ceph-mon[126353]: from='osd.7 [v2:172.21.15.158:6816/2578758635,v1:172.21.15.158:6817/2578758635]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]: dispatch 2023-07-20T02:49:07.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:07 smithi007 ceph-mon[126353]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]: dispatch 2023-07-20T02:49:07.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:07 smithi007 ceph-mon[126353]: Detected new or changed devices on smithi158 2023-07-20T02:49:07.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:07 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:07.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:07 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:07.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:07 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:49:07.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:07 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:49:07.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:07 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:49:07.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:07 smithi007 ceph-mon[126353]: Adjusting osd_memory_target on smithi158 to 5722M 2023-07-20T02:49:07.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:07 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:07.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:07 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:49:07.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:07 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:49:07.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:07 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:07.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:07 smithi007 ceph-mon[126353]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:49:07.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:07 smithi188 ceph-mon[129553]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-20T02:49:07.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:07 smithi188 ceph-mon[129553]: osdmap e48: 8 total, 7 up, 8 in 2023-07-20T02:49:07.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:07 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:49:07.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:07 smithi188 ceph-mon[129553]: from='osd.7 [v2:172.21.15.158:6816/2578758635,v1:172.21.15.158:6817/2578758635]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]: dispatch 2023-07-20T02:49:07.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:07 smithi188 ceph-mon[129553]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]: dispatch 2023-07-20T02:49:07.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:07 smithi188 ceph-mon[129553]: Detected new or changed devices on smithi158 2023-07-20T02:49:07.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:07 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:07.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:07 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:07.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:07 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:49:07.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:07 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:49:07.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:07 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:49:07.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:07 smithi188 ceph-mon[129553]: Adjusting osd_memory_target on smithi158 to 5722M 2023-07-20T02:49:07.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:07 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:07.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:07 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:49:07.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:07 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:49:07.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:07 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:07.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:07 smithi188 ceph-mon[129553]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:49:08.139 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:49:07 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:49:07.683+0000 7f6a1da81700 -1 osd.7 0 waiting for initial osdmap 2023-07-20T02:49:08.139 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:49:07 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:49:07.691+0000 7f6a1586f700 -1 osd.7 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T02:49:08.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:07 smithi158 ceph-mon[128925]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-20T02:49:08.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:07 smithi158 ceph-mon[128925]: osdmap e48: 8 total, 7 up, 8 in 2023-07-20T02:49:08.140 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:07 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:49:08.140 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:07 smithi158 ceph-mon[128925]: from='osd.7 [v2:172.21.15.158:6816/2578758635,v1:172.21.15.158:6817/2578758635]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]: dispatch 2023-07-20T02:49:08.140 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:07 smithi158 ceph-mon[128925]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]: dispatch 2023-07-20T02:49:08.140 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:07 smithi158 ceph-mon[128925]: Detected new or changed devices on smithi158 2023-07-20T02:49:08.140 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:07 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:08.140 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:07 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:08.140 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:07 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:49:08.141 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:07 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:49:08.141 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:07 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:49:08.141 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:07 smithi158 ceph-mon[128925]: Adjusting osd_memory_target on smithi158 to 5722M 2023-07-20T02:49:08.141 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:07 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:08.141 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:07 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:49:08.141 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:07 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:49:08.141 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:07 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:08.141 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:07 smithi158 ceph-mon[128925]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:49:08.294 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T02:49:08.929 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:08 smithi188 ceph-mon[129553]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]': finished 2023-07-20T02:49:08.930 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:08 smithi188 ceph-mon[129553]: osdmap e49: 8 total, 7 up, 8 in 2023-07-20T02:49:08.930 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:08 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:49:08.930 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:08 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:49:08.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:08 smithi007 ceph-mon[126353]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]': finished 2023-07-20T02:49:08.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:08 smithi007 ceph-mon[126353]: osdmap e49: 8 total, 7 up, 8 in 2023-07-20T02:49:08.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:08 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:49:08.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:08 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:49:09.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:08 smithi158 ceph-mon[128925]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]': finished 2023-07-20T02:49:09.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:08 smithi158 ceph-mon[128925]: osdmap e49: 8 total, 7 up, 8 in 2023-07-20T02:49:09.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:08 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:49:09.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:08 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:49:09.646 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.634+0000 7fd3f2462700 1 -- 172.21.15.188:0/2854241746 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd3ec100900 msgr2=0x7fd3ec100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:49:09.647 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.634+0000 7fd3f2462700 1 --2- 172.21.15.188:0/2854241746 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd3ec100900 0x7fd3ec100d60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc009990 tx=0x7fd3dc009ca0 comp rx=0 tx=0).stop 2023-07-20T02:49:09.647 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.634+0000 7fd3f2462700 1 -- 172.21.15.188:0/2854241746 shutdown_connections 2023-07-20T02:49:09.647 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.634+0000 7fd3f2462700 1 --2- 172.21.15.188:0/2854241746 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3ec1012a0 0x7fd3ec107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:49:09.647 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.634+0000 7fd3f2462700 1 --2- 172.21.15.188:0/2854241746 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3ec0ff700 0x7fd3ec0ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:49:09.647 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.634+0000 7fd3f2462700 1 --2- 172.21.15.188:0/2854241746 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd3ec100900 0x7fd3ec100d60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:49:09.648 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.634+0000 7fd3f2462700 1 -- 172.21.15.188:0/2854241746 >> 172.21.15.188:0/2854241746 conn(0x7fd3ec0fac80 msgr2=0x7fd3ec0fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:49:09.648 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.634+0000 7fd3f2462700 1 -- 172.21.15.188:0/2854241746 shutdown_connections 2023-07-20T02:49:09.648 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.634+0000 7fd3f2462700 1 -- 172.21.15.188:0/2854241746 wait complete. 2023-07-20T02:49:09.648 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.635+0000 7fd3f2462700 1 Processor -- start 2023-07-20T02:49:09.648 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.635+0000 7fd3f2462700 1 -- start start 2023-07-20T02:49:09.650 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.635+0000 7fd3f2462700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3ec0ff700 0x7fd3ec068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:49:09.651 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.635+0000 7fd3f2462700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3ec100900 0x7fd3ec19b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:49:09.651 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.635+0000 7fd3f2462700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd3ec1012a0 0x7fd3ec19d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:49:09.651 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.635+0000 7fd3f2462700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ec19dfa0 con 0x7fd3ec100900 2023-07-20T02:49:09.651 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.635+0000 7fd3f2462700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ec19e110 con 0x7fd3ec0ff700 2023-07-20T02:49:09.652 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.635+0000 7fd3f2462700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ec19e2d0 con 0x7fd3ec1012a0 2023-07-20T02:49:09.652 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.635+0000 7fd3f09ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd3ec1012a0 0x7fd3ec19d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:49:09.652 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.635+0000 7fd3f09ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd3ec1012a0 0x7fd3ec19d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:35054/0 (socket says 172.21.15.188:35054) 2023-07-20T02:49:09.652 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.635+0000 7fd3f09ff700 1 -- 172.21.15.188:0/3382742567 learned_addr learned my addr 172.21.15.188:0/3382742567 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-07-20T02:49:09.652 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.635+0000 7fd3ebfff700 1 --2- 172.21.15.188:0/3382742567 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3ec0ff700 0x7fd3ec068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:49:09.652 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.635+0000 7fd3eb7fe700 1 --2- 172.21.15.188:0/3382742567 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3ec100900 0x7fd3ec19b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:49:09.653 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.635+0000 7fd3f09ff700 1 -- 172.21.15.188:0/3382742567 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3ec0ff700 msgr2=0x7fd3ec068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:49:09.653 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.635+0000 7fd3f09ff700 1 --2- 172.21.15.188:0/3382742567 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3ec0ff700 0x7fd3ec068db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:49:09.653 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.635+0000 7fd3f09ff700 1 -- 172.21.15.188:0/3382742567 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3ec100900 msgr2=0x7fd3ec19b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:49:09.653 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.636+0000 7fd3f09ff700 1 --2- 172.21.15.188:0/3382742567 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3ec100900 0x7fd3ec19b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:49:09.653 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.636+0000 7fd3f09ff700 1 -- 172.21.15.188:0/3382742567 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3dc0095f0 con 0x7fd3ec1012a0 2023-07-20T02:49:09.653 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.636+0000 7fd3eb7fe700 1 --2- 172.21.15.188:0/3382742567 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3ec100900 0x7fd3ec19b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T02:49:09.653 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.636+0000 7fd3f09ff700 1 --2- 172.21.15.188:0/3382742567 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd3ec1012a0 0x7fd3ec19d800 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd3e4007ee0 tx=0x7fd3e400d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:49:09.653 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.636+0000 7fd3e97fa700 1 -- 172.21.15.188:0/3382742567 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e4017070 con 0x7fd3ec1012a0 2023-07-20T02:49:09.654 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.636+0000 7fd3e97fa700 1 -- 172.21.15.188:0/3382742567 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd3e400f040 con 0x7fd3ec1012a0 2023-07-20T02:49:09.654 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.636+0000 7fd3f2462700 1 -- 172.21.15.188:0/3382742567 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3ec19e5b0 con 0x7fd3ec1012a0 2023-07-20T02:49:09.654 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.636+0000 7fd3e97fa700 1 -- 172.21.15.188:0/3382742567 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e4005740 con 0x7fd3ec1012a0 2023-07-20T02:49:09.654 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.636+0000 7fd3f2462700 1 -- 172.21.15.188:0/3382742567 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3ec19ea60 con 0x7fd3ec1012a0 2023-07-20T02:49:09.654 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.637+0000 7fd3f2462700 1 -- 172.21.15.188:0/3382742567 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3ec10af00 con 0x7fd3ec1012a0 2023-07-20T02:49:09.654 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.638+0000 7fd3e97fa700 1 -- 172.21.15.188:0/3382742567 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fd3e400dc60 con 0x7fd3ec1012a0 2023-07-20T02:49:09.655 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.638+0000 7fd3e97fa700 1 --2- 172.21.15.188:0/3382742567 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fd3d40a1150 0x7fd3d40a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:49:09.655 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.639+0000 7fd3e97fa700 1 -- 172.21.15.188:0/3382742567 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5430+0+0 (secure 0 0 0) 0x7fd3e40cb610 con 0x7fd3ec1012a0 2023-07-20T02:49:09.656 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.639+0000 7fd3ebfff700 1 --2- 172.21.15.188:0/3382742567 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fd3d40a1150 0x7fd3d40a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:49:09.656 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.639+0000 7fd3ebfff700 1 --2- 172.21.15.188:0/3382742567 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fd3d40a1150 0x7fd3d40a3610 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec100760 tx=0x7fd3e000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:49:09.656 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.643+0000 7fd3e97fa700 1 -- 172.21.15.188:0/3382742567 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd3e40958a0 con 0x7fd3ec1012a0 2023-07-20T02:49:09.849 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:09.837+0000 7fd3f2462700 1 -- 172.21.15.188:0/3382742567 --> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fd3ec060bb0 con 0x7fd3d40a1150 2023-07-20T02:49:09.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:09 smithi007 ceph-mon[126353]: purged_snaps scrub starts 2023-07-20T02:49:09.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:09 smithi007 ceph-mon[126353]: purged_snaps scrub ok 2023-07-20T02:49:09.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:09 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:49:09.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:09 smithi007 ceph-mon[126353]: osd.7 [v2:172.21.15.158:6816/2578758635,v1:172.21.15.158:6817/2578758635] boot 2023-07-20T02:49:09.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:09 smithi007 ceph-mon[126353]: osdmap e50: 8 total, 8 up, 8 in 2023-07-20T02:49:09.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:09 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:49:09.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:09 smithi007 ceph-mon[126353]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:49:09.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:09 smithi188 ceph-mon[129553]: purged_snaps scrub starts 2023-07-20T02:49:09.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:09 smithi188 ceph-mon[129553]: purged_snaps scrub ok 2023-07-20T02:49:09.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:09 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:49:09.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:09 smithi188 ceph-mon[129553]: osd.7 [v2:172.21.15.158:6816/2578758635,v1:172.21.15.158:6817/2578758635] boot 2023-07-20T02:49:09.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:09 smithi188 ceph-mon[129553]: osdmap e50: 8 total, 8 up, 8 in 2023-07-20T02:49:09.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:09 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:49:09.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:09 smithi188 ceph-mon[129553]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:49:10.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:09 smithi158 ceph-mon[128925]: purged_snaps scrub starts 2023-07-20T02:49:10.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:09 smithi158 ceph-mon[128925]: purged_snaps scrub ok 2023-07-20T02:49:10.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:09 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:49:10.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:09 smithi158 ceph-mon[128925]: osd.7 [v2:172.21.15.158:6816/2578758635,v1:172.21.15.158:6817/2578758635] boot 2023-07-20T02:49:10.140 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:09 smithi158 ceph-mon[128925]: osdmap e50: 8 total, 8 up, 8 in 2023-07-20T02:49:10.140 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:09 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T02:49:10.140 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:09 smithi158 ceph-mon[128925]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-20T02:49:10.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:10 smithi188 ceph-mon[129553]: osdmap e51: 8 total, 8 up, 8 in 2023-07-20T02:49:10.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:10 smithi188 ceph-mon[129553]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:49:10.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:10 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:49:10.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:10 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:49:10.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:10 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:49:11.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:10 smithi158 ceph-mon[128925]: osdmap e51: 8 total, 8 up, 8 in 2023-07-20T02:49:11.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:10 smithi158 ceph-mon[128925]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:49:11.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:10 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:49:11.140 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:10 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:49:11.140 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:10 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:49:11.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:10 smithi007 ceph-mon[126353]: osdmap e51: 8 total, 8 up, 8 in 2023-07-20T02:49:11.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:10 smithi007 ceph-mon[126353]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:49:11.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:10 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:49:11.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:10 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:49:11.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:10 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:49:11.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:11 smithi188 ceph-mon[129553]: osdmap e52: 8 total, 8 up, 8 in 2023-07-20T02:49:11.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:11 smithi188 ceph-mon[129553]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:49:12.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:11 smithi158 ceph-mon[128925]: osdmap e52: 8 total, 8 up, 8 in 2023-07-20T02:49:12.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:11 smithi158 ceph-mon[128925]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:49:12.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:11 smithi007 ceph-mon[126353]: osdmap e52: 8 total, 8 up, 8 in 2023-07-20T02:49:12.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:11 smithi007 ceph-mon[126353]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:49:13.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:12 smithi158 ceph-mon[128925]: osdmap e53: 8 total, 8 up, 8 in 2023-07-20T02:49:13.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:12 smithi158 ceph-mon[128925]: from='client.? 172.21.15.188:0/1251612445' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4081496a-45ee-4b25-8f17-a23665521187"}]: dispatch 2023-07-20T02:49:13.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:12 smithi158 ceph-mon[128925]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4081496a-45ee-4b25-8f17-a23665521187"}]: dispatch 2023-07-20T02:49:13.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:12 smithi158 ceph-mon[128925]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4081496a-45ee-4b25-8f17-a23665521187"}]': finished 2023-07-20T02:49:13.140 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:12 smithi158 ceph-mon[128925]: osdmap e54: 9 total, 8 up, 9 in 2023-07-20T02:49:13.140 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:12 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T02:49:13.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:12 smithi007 ceph-mon[126353]: osdmap e53: 8 total, 8 up, 8 in 2023-07-20T02:49:13.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:12 smithi007 ceph-mon[126353]: from='client.? 172.21.15.188:0/1251612445' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4081496a-45ee-4b25-8f17-a23665521187"}]: dispatch 2023-07-20T02:49:13.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:12 smithi007 ceph-mon[126353]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4081496a-45ee-4b25-8f17-a23665521187"}]: dispatch 2023-07-20T02:49:13.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:12 smithi007 ceph-mon[126353]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4081496a-45ee-4b25-8f17-a23665521187"}]': finished 2023-07-20T02:49:13.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:12 smithi007 ceph-mon[126353]: osdmap e54: 9 total, 8 up, 9 in 2023-07-20T02:49:13.198 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:12 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T02:49:13.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:12 smithi188 ceph-mon[129553]: osdmap e53: 8 total, 8 up, 8 in 2023-07-20T02:49:13.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:12 smithi188 ceph-mon[129553]: from='client.? 172.21.15.188:0/1251612445' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4081496a-45ee-4b25-8f17-a23665521187"}]: dispatch 2023-07-20T02:49:13.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:12 smithi188 ceph-mon[129553]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4081496a-45ee-4b25-8f17-a23665521187"}]: dispatch 2023-07-20T02:49:13.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:12 smithi188 ceph-mon[129553]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4081496a-45ee-4b25-8f17-a23665521187"}]': finished 2023-07-20T02:49:13.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:12 smithi188 ceph-mon[129553]: osdmap e54: 9 total, 8 up, 9 in 2023-07-20T02:49:13.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:12 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T02:49:14.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:13 smithi158 ceph-mon[128925]: from='client.? 172.21.15.188:0/353907521' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:49:14.139 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:13 smithi158 ceph-mon[128925]: pgmap v174: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:49:14.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:13 smithi007 ceph-mon[126353]: from='client.? 172.21.15.188:0/353907521' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:49:14.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:13 smithi007 ceph-mon[126353]: pgmap v174: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:49:14.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:13 smithi188 ceph-mon[129553]: from='client.? 172.21.15.188:0/353907521' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:49:14.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:13 smithi188 ceph-mon[129553]: pgmap v174: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:49:16.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:16 smithi188 ceph-mon[129553]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:49:16.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:16 smithi158 ceph-mon[128925]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:49:16.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:16 smithi007 ceph-mon[126353]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:49:18.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:18 smithi158 ceph-mon[128925]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:49:18.407 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:18 smithi188 ceph-mon[129553]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:49:18.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:18 smithi007 ceph-mon[126353]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:49:19.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:19 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-20T02:49:19.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:19 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:49:19.440 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:19 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-20T02:49:19.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:19 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:49:19.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:19 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-20T02:49:19.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:19 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:49:20.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:20 smithi188 ceph-mon[129553]: Deploying daemon osd.8 on smithi188 2023-07-20T02:49:20.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:20 smithi188 ceph-mon[129553]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:49:20.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:20 smithi158 ceph-mon[128925]: Deploying daemon osd.8 on smithi188 2023-07-20T02:49:20.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:20 smithi158 ceph-mon[128925]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:49:20.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:20 smithi007 ceph-mon[126353]: Deploying daemon osd.8 on smithi188 2023-07-20T02:49:20.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:20 smithi007 ceph-mon[126353]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:49:22.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:22 smithi158 ceph-mon[128925]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:49:22.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:22 smithi007 ceph-mon[126353]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:49:22.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:22 smithi188 ceph-mon[129553]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:49:24.692 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:24 smithi188 ceph-mon[129553]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:49:24.693 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:24 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:24.693 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:24 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:49:24.693 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:24 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:24.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:24 smithi158 ceph-mon[128925]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:49:24.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:24 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:24.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:24 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:49:24.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:24 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:24.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:24 smithi007 ceph-mon[126353]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:49:24.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:24 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:24.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:24 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:49:24.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:24 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:26.466 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:26.454+0000 7fd3e97fa700 1 -- 172.21.15.188:0/3382742567 <== mgr.14152 v2:172.21.15.7:6800/1354542137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd3ec060bb0 con 0x7fd3d40a1150 2023-07-20T02:49:26.466 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 8 on host 'smithi188' 2023-07-20T02:49:26.468 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:26.456+0000 7fd3f2462700 1 -- 172.21.15.188:0/3382742567 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fd3d40a1150 msgr2=0x7fd3d40a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:49:26.468 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:26.456+0000 7fd3f2462700 1 --2- 172.21.15.188:0/3382742567 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fd3d40a1150 0x7fd3d40a3610 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec100760 tx=0x7fd3e000b410 comp rx=0 tx=0).stop 2023-07-20T02:49:26.468 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:26.456+0000 7fd3f2462700 1 -- 172.21.15.188:0/3382742567 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd3ec1012a0 msgr2=0x7fd3ec19d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:49:26.468 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:26.456+0000 7fd3f2462700 1 --2- 172.21.15.188:0/3382742567 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd3ec1012a0 0x7fd3ec19d800 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd3e4007ee0 tx=0x7fd3e400d3b0 comp rx=0 tx=0).stop 2023-07-20T02:49:26.468 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:26.456+0000 7fd3f2462700 1 -- 172.21.15.188:0/3382742567 shutdown_connections 2023-07-20T02:49:26.469 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:26.456+0000 7fd3f2462700 1 --2- 172.21.15.188:0/3382742567 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd3ec1012a0 0x7fd3ec19d800 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:49:26.469 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:26.456+0000 7fd3f2462700 1 --2- 172.21.15.188:0/3382742567 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd3ec0ff700 0x7fd3ec068db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:49:26.469 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:26.456+0000 7fd3f2462700 1 --2- 172.21.15.188:0/3382742567 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fd3d40a1150 0x7fd3d40a3610 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:49:26.469 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:26.456+0000 7fd3f2462700 1 --2- 172.21.15.188:0/3382742567 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3ec100900 0x7fd3ec19b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:49:26.469 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:26.456+0000 7fd3f2462700 1 -- 172.21.15.188:0/3382742567 >> 172.21.15.188:0/3382742567 conn(0x7fd3ec0fac80 msgr2=0x7fd3ec0fcf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:49:26.469 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:26.456+0000 7fd3f2462700 1 -- 172.21.15.188:0/3382742567 shutdown_connections 2023-07-20T02:49:26.469 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:49:26.456+0000 7fd3f2462700 1 -- 172.21.15.188:0/3382742567 wait complete. 2023-07-20T02:49:26.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:26 smithi188 ceph-mon[129553]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:49:26.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:26 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:26.712 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:26 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:26.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:26 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:49:26.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:26 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:49:26.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:26 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:26.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:26 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:26.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:26 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:26.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:26 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:49:26.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:26 smithi158 ceph-mon[128925]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:49:26.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:26 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:26.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:26 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:26.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:26 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:49:26.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:26 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:49:26.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:26 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:26.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:26 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:26.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:26 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:26.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:26 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:49:26.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:26 smithi007 ceph-mon[126353]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:49:26.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:26 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:26.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:26 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:26.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:26 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:49:26.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:26 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:49:26.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:26 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:26.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:26 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:26.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:26 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:26.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:26 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:49:26.995 DEBUG:teuthology.orchestra.run.smithi188:osd.8> sudo journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.8.service 2023-07-20T02:49:26.997 INFO:tasks.cephadm:Deploying osd.9 on smithi007 with /dev/vg_nvme/lv_1... 2023-07-20T02:49:26.998 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-20T02:49:27.027 INFO:journalctl@ceph.osd.8.smithi188.stdout:-- Logs begin at Thu 2023-07-20 02:09:43 UTC. -- 2023-07-20T02:49:27.712 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:49:27 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:49:27.447+0000 7f6097e7c640 -1 osd.8 0 log_to_monitors true 2023-07-20T02:49:27.740 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:49:28.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:28 smithi158 ceph-mon[128925]: from='osd.8 [v2:172.21.15.188:6816/3020824843,v1:172.21.15.188:6817/3020824843]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T02:49:28.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:28 smithi158 ceph-mon[128925]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T02:49:28.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:28 smithi158 ceph-mon[128925]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:49:28.413 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:28 smithi007 ceph-mon[126353]: from='osd.8 [v2:172.21.15.188:6816/3020824843,v1:172.21.15.188:6817/3020824843]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T02:49:28.414 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:28 smithi007 ceph-mon[126353]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T02:49:28.414 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:28 smithi007 ceph-mon[126353]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:49:28.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:28 smithi188 ceph-mon[129553]: from='osd.8 [v2:172.21.15.188:6816/3020824843,v1:172.21.15.188:6817/3020824843]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T02:49:28.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:28 smithi188 ceph-mon[129553]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T02:49:28.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:28 smithi188 ceph-mon[129553]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:49:29.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:29 smithi158 ceph-mon[128925]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-20T02:49:29.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:29 smithi158 ceph-mon[128925]: osdmap e55: 9 total, 8 up, 9 in 2023-07-20T02:49:29.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:29 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T02:49:29.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:29 smithi158 ceph-mon[128925]: from='osd.8 [v2:172.21.15.188:6816/3020824843,v1:172.21.15.188:6817/3020824843]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-07-20T02:49:29.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:29 smithi158 ceph-mon[128925]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-07-20T02:49:29.446 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:29 smithi188 ceph-mon[129553]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-20T02:49:29.446 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:29 smithi188 ceph-mon[129553]: osdmap e55: 9 total, 8 up, 9 in 2023-07-20T02:49:29.446 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:29 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T02:49:29.446 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:29 smithi188 ceph-mon[129553]: from='osd.8 [v2:172.21.15.188:6816/3020824843,v1:172.21.15.188:6817/3020824843]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-07-20T02:49:29.447 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:29 smithi188 ceph-mon[129553]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-07-20T02:49:29.447 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:49:29 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:49:29.116+0000 7f608d4f4700 -1 osd.8 0 waiting for initial osdmap 2023-07-20T02:49:29.447 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:49:29 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:49:29.129+0000 7f60862e4700 -1 osd.8 56 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T02:49:29.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:29 smithi007 ceph-mon[126353]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-20T02:49:29.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:29 smithi007 ceph-mon[126353]: osdmap e55: 9 total, 8 up, 9 in 2023-07-20T02:49:29.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:29 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T02:49:29.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:29 smithi007 ceph-mon[126353]: from='osd.8 [v2:172.21.15.188:6816/3020824843,v1:172.21.15.188:6817/3020824843]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-07-20T02:49:29.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:29 smithi007 ceph-mon[126353]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-07-20T02:49:29.854 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-20T02:49:29.854 INFO:teuthology.orchestra.run.smithi007.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-20T02:49:29.854 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10+0 records in 2023-07-20T02:49:29.854 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2023-07-20T02:49:29.854 INFO:teuthology.orchestra.run.smithi007.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0432852 s, 242 MB/s 2023-07-20T02:49:29.854 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 2023-07-20T02:49:29.855 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping successful for: 2023-07-20T02:49:30.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:30 smithi007 ceph-mon[126353]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2023-07-20T02:49:30.154 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:30 smithi007 ceph-mon[126353]: osdmap e56: 9 total, 8 up, 9 in 2023-07-20T02:49:30.154 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:30 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T02:49:30.154 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:30 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T02:49:30.154 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:30 smithi007 ceph-mon[126353]: Detected new or changed devices on smithi188 2023-07-20T02:49:30.154 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:30 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:30.154 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:30 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:30.154 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:30 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:49:30.155 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:30 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:49:30.155 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:30 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:49:30.155 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:30 smithi007 ceph-mon[126353]: Adjusting osd_memory_target on smithi188 to 5722M 2023-07-20T02:49:30.155 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:30 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:30.155 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:30 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:49:30.155 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:30 smithi007 ceph-mon[126353]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:49:30.155 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:30 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:49:30.156 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:30 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:30.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:30 smithi158 ceph-mon[128925]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2023-07-20T02:49:30.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:30 smithi158 ceph-mon[128925]: osdmap e56: 9 total, 8 up, 9 in 2023-07-20T02:49:30.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:30 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T02:49:30.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:30 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T02:49:30.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:30 smithi158 ceph-mon[128925]: Detected new or changed devices on smithi188 2023-07-20T02:49:30.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:30 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:30.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:30 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:30.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:30 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:49:30.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:30 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:49:30.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:30 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:49:30.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:30 smithi158 ceph-mon[128925]: Adjusting osd_memory_target on smithi188 to 5722M 2023-07-20T02:49:30.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:30 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:30.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:30 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:49:30.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:30 smithi158 ceph-mon[128925]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:49:30.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:30 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:49:30.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:30 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:30.416 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi007:vg_nvme/lv_1 2023-07-20T02:49:30.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:30 smithi188 ceph-mon[129553]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2023-07-20T02:49:30.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:30 smithi188 ceph-mon[129553]: osdmap e56: 9 total, 8 up, 9 in 2023-07-20T02:49:30.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:30 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T02:49:30.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:30 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T02:49:30.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:30 smithi188 ceph-mon[129553]: Detected new or changed devices on smithi188 2023-07-20T02:49:30.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:30 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:30.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:30 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:30.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:30 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:49:30.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:30 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:49:30.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:30 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:49:30.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:30 smithi188 ceph-mon[129553]: Adjusting osd_memory_target on smithi188 to 5722M 2023-07-20T02:49:30.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:30 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:30.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:30 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:49:30.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:30 smithi188 ceph-mon[129553]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-20T02:49:30.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:30 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:49:30.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:30 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:31.149 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:49:31.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:31 smithi158 ceph-mon[128925]: purged_snaps scrub starts 2023-07-20T02:49:31.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:31 smithi158 ceph-mon[128925]: purged_snaps scrub ok 2023-07-20T02:49:31.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:31 smithi158 ceph-mon[128925]: osd.8 [v2:172.21.15.188:6816/3020824843,v1:172.21.15.188:6817/3020824843] boot 2023-07-20T02:49:31.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:31 smithi158 ceph-mon[128925]: osdmap e57: 9 total, 9 up, 9 in 2023-07-20T02:49:31.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:31 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T02:49:31.414 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:31 smithi007 ceph-mon[126353]: purged_snaps scrub starts 2023-07-20T02:49:31.414 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:31 smithi007 ceph-mon[126353]: purged_snaps scrub ok 2023-07-20T02:49:31.415 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:31 smithi007 ceph-mon[126353]: osd.8 [v2:172.21.15.188:6816/3020824843,v1:172.21.15.188:6817/3020824843] boot 2023-07-20T02:49:31.415 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:31 smithi007 ceph-mon[126353]: osdmap e57: 9 total, 9 up, 9 in 2023-07-20T02:49:31.415 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:31 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T02:49:31.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:31 smithi188 ceph-mon[129553]: purged_snaps scrub starts 2023-07-20T02:49:31.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:31 smithi188 ceph-mon[129553]: purged_snaps scrub ok 2023-07-20T02:49:31.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:31 smithi188 ceph-mon[129553]: osd.8 [v2:172.21.15.188:6816/3020824843,v1:172.21.15.188:6817/3020824843] boot 2023-07-20T02:49:31.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:31 smithi188 ceph-mon[129553]: osdmap e57: 9 total, 9 up, 9 in 2023-07-20T02:49:31.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:31 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T02:49:32.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:32 smithi158 ceph-mon[128925]: osdmap e58: 9 total, 9 up, 9 in 2023-07-20T02:49:32.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:32 smithi158 ceph-mon[128925]: pgmap v187: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:49:32.433 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:32 smithi007 ceph-mon[126353]: osdmap e58: 9 total, 9 up, 9 in 2023-07-20T02:49:32.434 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:32 smithi007 ceph-mon[126353]: pgmap v187: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:49:32.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:32 smithi188 ceph-mon[129553]: osdmap e58: 9 total, 9 up, 9 in 2023-07-20T02:49:32.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:32 smithi188 ceph-mon[129553]: pgmap v187: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:49:32.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.669+0000 7f0ed7730700 1 -- 172.21.15.7:0/3435150794 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ed0101a00 msgr2=0x7f0ed0101e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:49:32.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.669+0000 7f0ed7730700 1 --2- 172.21.15.7:0/3435150794 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ed0101a00 0x7f0ed0101e60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0ec00099c0 tx=0x7f0ec0009cd0 comp rx=0 tx=0).stop 2023-07-20T02:49:32.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.669+0000 7f0ed7730700 1 -- 172.21.15.7:0/3435150794 shutdown_connections 2023-07-20T02:49:32.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.669+0000 7f0ed7730700 1 --2- 172.21.15.7:0/3435150794 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ed01023a0 0x7f0ed0108c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:49:32.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.669+0000 7f0ed7730700 1 --2- 172.21.15.7:0/3435150794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ed0100800 0x7f0ed0100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:49:32.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.669+0000 7f0ed7730700 1 --2- 172.21.15.7:0/3435150794 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ed0101a00 0x7f0ed0101e60 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:49:32.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.669+0000 7f0ed7730700 1 -- 172.21.15.7:0/3435150794 >> 172.21.15.7:0/3435150794 conn(0x7f0ed00fbd60 msgr2=0x7f0ed00fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:49:32.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.669+0000 7f0ed7730700 1 -- 172.21.15.7:0/3435150794 shutdown_connections 2023-07-20T02:49:32.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.669+0000 7f0ed7730700 1 -- 172.21.15.7:0/3435150794 wait complete. 2023-07-20T02:49:32.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.670+0000 7f0ed7730700 1 Processor -- start 2023-07-20T02:49:32.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.670+0000 7f0ed7730700 1 -- start start 2023-07-20T02:49:32.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.670+0000 7f0ed7730700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ed0100800 0x7f0ed0196490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:49:32.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.670+0000 7f0ed7730700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ed0101a00 0x7f0ed01969d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:49:32.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.670+0000 7f0ed7730700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ed01023a0 0x7f0ed019da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:49:32.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.670+0000 7f0ed7730700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed019e020 con 0x7f0ed0101a00 2023-07-20T02:49:32.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.670+0000 7f0ed7730700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed019e160 con 0x7f0ed0100800 2023-07-20T02:49:32.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.670+0000 7f0ed7730700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed019e2f0 con 0x7f0ed01023a0 2023-07-20T02:49:32.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.671+0000 7f0ed54cc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ed0100800 0x7f0ed0196490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:49:32.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.671+0000 7f0ed4ccb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ed0101a00 0x7f0ed01969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:49:32.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.671+0000 7f0ed54cc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ed0100800 0x7f0ed0196490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.7:49440/0 (socket says 172.21.15.7:49440) 2023-07-20T02:49:32.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.671+0000 7f0ed54cc700 1 -- 172.21.15.7:0/1955111430 learned_addr learned my addr 172.21.15.7:0/1955111430 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:49:32.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.671+0000 7f0ed5ccd700 1 --2- 172.21.15.7:0/1955111430 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ed01023a0 0x7f0ed019da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:49:32.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.671+0000 7f0ed54cc700 1 -- 172.21.15.7:0/1955111430 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ed01023a0 msgr2=0x7f0ed019da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:49:32.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.671+0000 7f0ed54cc700 1 --2- 172.21.15.7:0/1955111430 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ed01023a0 0x7f0ed019da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:49:32.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.671+0000 7f0ed54cc700 1 -- 172.21.15.7:0/1955111430 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ed0101a00 msgr2=0x7f0ed01969d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:49:32.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.671+0000 7f0ed54cc700 1 --2- 172.21.15.7:0/1955111430 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ed0101a00 0x7f0ed01969d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:49:32.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.671+0000 7f0ed54cc700 1 -- 172.21.15.7:0/1955111430 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec00095f0 con 0x7f0ed0100800 2023-07-20T02:49:32.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.671+0000 7f0ed4ccb700 1 --2- 172.21.15.7:0/1955111430 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ed0101a00 0x7f0ed01969d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T02:49:32.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.671+0000 7f0ed54cc700 1 --2- 172.21.15.7:0/1955111430 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ed0100800 0x7f0ed0196490 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc00e970 tx=0x7f0ecc00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:49:32.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.672+0000 7f0ec67fc700 1 -- 172.21.15.7:0/1955111430 <== mon.2 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc009790 con 0x7f0ed0100800 2023-07-20T02:49:32.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.672+0000 7f0ec67fc700 1 -- 172.21.15.7:0/1955111430 <== mon.2 v2:172.21.15.158:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0ecc00c7e0 con 0x7f0ed0100800 2023-07-20T02:49:32.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.672+0000 7f0ed7730700 1 -- 172.21.15.7:0/1955111430 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ed019e5d0 con 0x7f0ed0100800 2023-07-20T02:49:32.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.672+0000 7f0ec67fc700 1 -- 172.21.15.7:0/1955111430 <== mon.2 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc003c70 con 0x7f0ed0100800 2023-07-20T02:49:32.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.672+0000 7f0ed7730700 1 -- 172.21.15.7:0/1955111430 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ed019eb20 con 0x7f0ed0100800 2023-07-20T02:49:32.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.673+0000 7f0ed7730700 1 -- 172.21.15.7:0/1955111430 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ed010c000 con 0x7f0ed0100800 2023-07-20T02:49:32.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.675+0000 7f0ec67fc700 1 -- 172.21.15.7:0/1955111430 <== mon.2 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f0ecc00c950 con 0x7f0ed0100800 2023-07-20T02:49:32.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.677+0000 7f0ec67fc700 1 --2- 172.21.15.7:0/1955111430 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f0ebc0a11d0 0x7f0ebc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:49:32.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.677+0000 7f0ec67fc700 1 -- 172.21.15.7:0/1955111430 <== mon.2 v2:172.21.15.158:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5858+0+0 (secure 0 0 0) 0x7f0ecc014070 con 0x7f0ed0100800 2023-07-20T02:49:32.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.677+0000 7f0ed4ccb700 1 --2- 172.21.15.7:0/1955111430 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f0ebc0a11d0 0x7f0ebc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:49:32.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.677+0000 7f0ed4ccb700 1 --2- 172.21.15.7:0/1955111430 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f0ebc0a11d0 0x7f0ebc0a3690 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0ec000b580 tx=0x7f0ec001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:49:32.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.679+0000 7f0ec67fc700 1 -- 172.21.15.7:0/1955111430 <== mon.2 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0ecc096320 con 0x7f0ed0100800 2023-07-20T02:49:32.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:32.892+0000 7f0ed7730700 1 -- 172.21.15.7:0/1955111430 --> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ed0060bb0 con 0x7f0ebc0a11d0 2023-07-20T02:49:33.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:33 smithi007 ceph-mon[126353]: osdmap e59: 9 total, 9 up, 9 in 2023-07-20T02:49:33.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:33 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:49:33.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:33 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:49:33.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:33 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:49:33.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:33 smithi188 ceph-mon[129553]: osdmap e59: 9 total, 9 up, 9 in 2023-07-20T02:49:33.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:33 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:49:33.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:33 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:49:33.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:33 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:49:33.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:33 smithi158 ceph-mon[128925]: osdmap e59: 9 total, 9 up, 9 in 2023-07-20T02:49:33.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:33 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:49:33.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:33 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:49:33.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:33 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:49:34.228 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:34 smithi007 ceph-mon[126353]: from='client.24353 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:49:34.229 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:34 smithi007 ceph-mon[126353]: osdmap e60: 9 total, 9 up, 9 in 2023-07-20T02:49:34.229 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:34 smithi007 ceph-mon[126353]: pgmap v190: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:49:34.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:34 smithi188 ceph-mon[129553]: from='client.24353 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:49:34.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:34 smithi188 ceph-mon[129553]: osdmap e60: 9 total, 9 up, 9 in 2023-07-20T02:49:34.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:34 smithi188 ceph-mon[129553]: pgmap v190: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:49:34.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:34 smithi158 ceph-mon[128925]: from='client.24353 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:49:34.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:34 smithi158 ceph-mon[128925]: osdmap e60: 9 total, 9 up, 9 in 2023-07-20T02:49:34.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:34 smithi158 ceph-mon[128925]: pgmap v190: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:49:35.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:35 smithi007 ceph-mon[126353]: osdmap e61: 9 total, 9 up, 9 in 2023-07-20T02:49:35.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:35 smithi188 ceph-mon[129553]: osdmap e61: 9 total, 9 up, 9 in 2023-07-20T02:49:35.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:35 smithi158 ceph-mon[128925]: osdmap e61: 9 total, 9 up, 9 in 2023-07-20T02:49:36.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:36 smithi007 ceph-mon[126353]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 131 KiB/s, 0 objects/s recovering 2023-07-20T02:49:36.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:36 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/1345443617' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "723580ae-efe3-4d86-95fd-d7a4c0631425"}]: dispatch 2023-07-20T02:49:36.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:36 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/1345443617' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "723580ae-efe3-4d86-95fd-d7a4c0631425"}]': finished 2023-07-20T02:49:36.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:36 smithi007 ceph-mon[126353]: osdmap e62: 10 total, 9 up, 10 in 2023-07-20T02:49:36.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:36 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T02:49:36.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:36 smithi188 ceph-mon[129553]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 131 KiB/s, 0 objects/s recovering 2023-07-20T02:49:36.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:36 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/1345443617' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "723580ae-efe3-4d86-95fd-d7a4c0631425"}]: dispatch 2023-07-20T02:49:36.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:36 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/1345443617' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "723580ae-efe3-4d86-95fd-d7a4c0631425"}]': finished 2023-07-20T02:49:36.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:36 smithi188 ceph-mon[129553]: osdmap e62: 10 total, 9 up, 10 in 2023-07-20T02:49:36.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:36 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T02:49:36.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:36 smithi158 ceph-mon[128925]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 131 KiB/s, 0 objects/s recovering 2023-07-20T02:49:36.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:36 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/1345443617' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "723580ae-efe3-4d86-95fd-d7a4c0631425"}]: dispatch 2023-07-20T02:49:36.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:36 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/1345443617' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "723580ae-efe3-4d86-95fd-d7a4c0631425"}]': finished 2023-07-20T02:49:36.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:36 smithi158 ceph-mon[128925]: osdmap e62: 10 total, 9 up, 10 in 2023-07-20T02:49:36.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:36 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T02:49:37.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:37 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/777808210' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:49:37.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:37 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/777808210' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:49:37.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:37 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/777808210' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:49:38.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:38 smithi007 ceph-mon[126353]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 107 KiB/s, 0 objects/s recovering 2023-07-20T02:49:38.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:38 smithi188 ceph-mon[129553]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 107 KiB/s, 0 objects/s recovering 2023-07-20T02:49:38.639 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:38 smithi158 ceph-mon[128925]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 107 KiB/s, 0 objects/s recovering 2023-07-20T02:49:40.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:40 smithi158 ceph-mon[128925]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 90 KiB/s, 0 objects/s recovering 2023-07-20T02:49:40.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:40 smithi007 ceph-mon[126353]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 90 KiB/s, 0 objects/s recovering 2023-07-20T02:49:40.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:40 smithi188 ceph-mon[129553]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 90 KiB/s, 0 objects/s recovering 2023-07-20T02:49:42.855 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:42 smithi007 ceph-mon[126353]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T02:49:42.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:42 smithi158 ceph-mon[128925]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T02:49:42.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:42 smithi188 ceph-mon[129553]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T02:49:43.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-20T02:49:43.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:43 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:49:43.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:43 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-20T02:49:43.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:43 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:49:43.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:43 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-20T02:49:43.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:43 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:49:44.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:44 smithi007 ceph-mon[126353]: Deploying daemon osd.9 on smithi007 2023-07-20T02:49:44.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:44 smithi007 ceph-mon[126353]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-07-20T02:49:44.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:44 smithi158 ceph-mon[128925]: Deploying daemon osd.9 on smithi007 2023-07-20T02:49:44.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:44 smithi158 ceph-mon[128925]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-07-20T02:49:44.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:44 smithi188 ceph-mon[129553]: Deploying daemon osd.9 on smithi007 2023-07-20T02:49:44.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:44 smithi188 ceph-mon[129553]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-07-20T02:49:46.690 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:46 smithi007 ceph-mon[126353]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:49:46.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:46 smithi158 ceph-mon[128925]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:49:46.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:46 smithi188 ceph-mon[129553]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:49:48.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:48 smithi158 ceph-mon[128925]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:49:48.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:48 smithi007 ceph-mon[126353]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:49:48.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:48 smithi188 ceph-mon[129553]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:49:50.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:50 smithi007 ceph-mon[126353]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:49:50.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:50 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:50.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:50 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:50.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:50 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:49:50.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:50 smithi158 ceph-mon[128925]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:49:50.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:50 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:50.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:50 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:50.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:50 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:49:50.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:50 smithi188 ceph-mon[129553]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:49:50.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:50 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:50.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:50 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:50.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:50 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:49:52.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:52 smithi007 ceph-mon[126353]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:49:52.889 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:52 smithi158 ceph-mon[128925]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:49:52.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:52 smithi188 ceph-mon[129553]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:49:53.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:53.553+0000 7f0ec67fc700 1 -- 172.21.15.7:0/1955111430 <== mgr.14152 v2:172.21.15.7:6800/1354542137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0ed0060bb0 con 0x7f0ebc0a11d0 2023-07-20T02:49:53.566 INFO:teuthology.orchestra.run.smithi007.stdout:Created osd(s) 9 on host 'smithi007' 2023-07-20T02:49:53.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:53.556+0000 7f0ed7730700 1 -- 172.21.15.7:0/1955111430 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f0ebc0a11d0 msgr2=0x7f0ebc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:49:53.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:53.556+0000 7f0ed7730700 1 --2- 172.21.15.7:0/1955111430 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f0ebc0a11d0 0x7f0ebc0a3690 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0ec000b580 tx=0x7f0ec001a040 comp rx=0 tx=0).stop 2023-07-20T02:49:53.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:53.556+0000 7f0ed7730700 1 -- 172.21.15.7:0/1955111430 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ed0100800 msgr2=0x7f0ed0196490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:49:53.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:53.556+0000 7f0ed7730700 1 --2- 172.21.15.7:0/1955111430 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ed0100800 0x7f0ed0196490 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc00e970 tx=0x7f0ecc00ec80 comp rx=0 tx=0).stop 2023-07-20T02:49:53.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:53.556+0000 7f0ed7730700 1 -- 172.21.15.7:0/1955111430 shutdown_connections 2023-07-20T02:49:53.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:53.556+0000 7f0ed7730700 1 --2- 172.21.15.7:0/1955111430 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ed01023a0 0x7f0ed019da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:49:53.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:53.556+0000 7f0ed7730700 1 --2- 172.21.15.7:0/1955111430 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f0ed0100800 0x7f0ed0196490 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:49:53.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:53.556+0000 7f0ed7730700 1 --2- 172.21.15.7:0/1955111430 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f0ebc0a11d0 0x7f0ebc0a3690 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:49:53.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:53.556+0000 7f0ed7730700 1 --2- 172.21.15.7:0/1955111430 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ed0101a00 0x7f0ed01969d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:49:53.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:53.556+0000 7f0ed7730700 1 -- 172.21.15.7:0/1955111430 >> 172.21.15.7:0/1955111430 conn(0x7f0ed00fbd60 msgr2=0x7f0ed00fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:49:53.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:53.556+0000 7f0ed7730700 1 -- 172.21.15.7:0/1955111430 shutdown_connections 2023-07-20T02:49:53.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:49:53.556+0000 7f0ed7730700 1 -- 172.21.15.7:0/1955111430 wait complete. 2023-07-20T02:49:53.988 DEBUG:teuthology.orchestra.run.smithi007:osd.9> sudo journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.9.service 2023-07-20T02:49:53.995 INFO:tasks.cephadm:Deploying osd.10 on smithi158 with /dev/vg_nvme/lv_1... 2023-07-20T02:49:53.996 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-20T02:49:54.068 INFO:journalctl@ceph.osd.9.smithi007.stdout:-- Logs begin at Thu 2023-07-20 02:22:32 UTC. -- 2023-07-20T02:49:54.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:53 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:54.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:53 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:54.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:53 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:49:54.198 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:53 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:49:54.198 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:53 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:54.198 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:53 smithi007 ceph-mon[126353]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:49:54.198 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:53 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:54.198 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:53 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:54.198 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:53 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:49:54.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:53 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:54.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:53 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:54.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:53 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:49:54.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:53 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:49:54.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:53 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:54.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:53 smithi188 ceph-mon[129553]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:49:54.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:53 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:54.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:53 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:54.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:53 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:49:54.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:53 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:54.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:53 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:54.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:53 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:49:54.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:53 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:49:54.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:53 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:54.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:53 smithi158 ceph-mon[128925]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:49:54.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:53 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:54.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:53 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:54.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:53 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:49:54.560 INFO:teuthology.orchestra.run.smithi158.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T02:49:55.055 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:54 smithi158 ceph-mon[128925]: from='osd.9 [v2:172.21.15.7:6826/1612464825,v1:172.21.15.7:6827/1612464825]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-20T02:49:55.055 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:54 smithi158 ceph-mon[128925]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-20T02:49:55.142 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:54 smithi007 ceph-mon[126353]: from='osd.9 [v2:172.21.15.7:6826/1612464825,v1:172.21.15.7:6827/1612464825]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-20T02:49:55.143 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:54 smithi007 ceph-mon[126353]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-20T02:49:55.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:54 smithi188 ceph-mon[129553]: from='osd.9 [v2:172.21.15.7:6826/1612464825,v1:172.21.15.7:6827/1612464825]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-20T02:49:55.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:54 smithi188 ceph-mon[129553]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-20T02:49:56.026 INFO:teuthology.orchestra.run.smithi158.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-20T02:49:56.026 INFO:teuthology.orchestra.run.smithi158.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-20T02:49:56.027 INFO:teuthology.orchestra.run.smithi158.stderr: stderr: 10+0 records in 2023-07-20T02:49:56.027 INFO:teuthology.orchestra.run.smithi158.stderr:10+0 records out 2023-07-20T02:49:56.028 INFO:teuthology.orchestra.run.smithi158.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00805802 s, 1.3 GB/s 2023-07-20T02:49:56.028 INFO:teuthology.orchestra.run.smithi158.stderr: stderr: 2023-07-20T02:49:56.028 INFO:teuthology.orchestra.run.smithi158.stderr:--> Zapping successful for: 2023-07-20T02:49:56.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:55 smithi007 ceph-mon[126353]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-20T02:49:56.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:55 smithi007 ceph-mon[126353]: osdmap e63: 10 total, 9 up, 10 in 2023-07-20T02:49:56.198 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:55 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T02:49:56.198 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:55 smithi007 ceph-mon[126353]: from='osd.9 [v2:172.21.15.7:6826/1612464825,v1:172.21.15.7:6827/1612464825]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-07-20T02:49:56.198 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:55 smithi007 ceph-mon[126353]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-07-20T02:49:56.198 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:55 smithi007 ceph-mon[126353]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:49:56.199 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:49:55 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:49:55.927+0000 7f8378572700 -1 osd.9 0 waiting for initial osdmap 2023-07-20T02:49:56.199 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:49:55 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:49:55.943+0000 7f8372364700 -1 osd.9 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T02:49:56.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:55 smithi188 ceph-mon[129553]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-20T02:49:56.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:55 smithi188 ceph-mon[129553]: osdmap e63: 10 total, 9 up, 10 in 2023-07-20T02:49:56.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:55 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T02:49:56.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:55 smithi188 ceph-mon[129553]: from='osd.9 [v2:172.21.15.7:6826/1612464825,v1:172.21.15.7:6827/1612464825]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-07-20T02:49:56.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:55 smithi188 ceph-mon[129553]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-07-20T02:49:56.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:55 smithi188 ceph-mon[129553]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:49:56.214 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:55 smithi158 ceph-mon[128925]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-20T02:49:56.215 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:55 smithi158 ceph-mon[128925]: osdmap e63: 10 total, 9 up, 10 in 2023-07-20T02:49:56.215 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:55 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T02:49:56.215 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:55 smithi158 ceph-mon[128925]: from='osd.9 [v2:172.21.15.7:6826/1612464825,v1:172.21.15.7:6827/1612464825]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-07-20T02:49:56.216 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:55 smithi158 ceph-mon[128925]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-07-20T02:49:56.216 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:55 smithi158 ceph-mon[128925]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-20T02:49:56.514 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi158:vg_nvme/lv_1 2023-07-20T02:49:57.069 INFO:teuthology.orchestra.run.smithi158.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T02:49:57.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:56 smithi007 ceph-mon[126353]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2023-07-20T02:49:57.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:56 smithi007 ceph-mon[126353]: osdmap e64: 10 total, 9 up, 10 in 2023-07-20T02:49:57.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:56 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T02:49:57.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:56 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T02:49:57.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:56 smithi188 ceph-mon[129553]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2023-07-20T02:49:57.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:56 smithi188 ceph-mon[129553]: osdmap e64: 10 total, 9 up, 10 in 2023-07-20T02:49:57.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:56 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T02:49:57.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:56 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T02:49:57.269 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:56 smithi158 ceph-mon[128925]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2023-07-20T02:49:57.270 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:56 smithi158 ceph-mon[128925]: osdmap e64: 10 total, 9 up, 10 in 2023-07-20T02:49:57.270 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:56 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T02:49:57.270 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:56 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T02:49:58.119 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:57 smithi158 ceph-mon[128925]: purged_snaps scrub starts 2023-07-20T02:49:58.119 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:57 smithi158 ceph-mon[128925]: purged_snaps scrub ok 2023-07-20T02:49:58.119 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:57 smithi158 ceph-mon[128925]: osd.9 [v2:172.21.15.7:6826/1612464825,v1:172.21.15.7:6827/1612464825] boot 2023-07-20T02:49:58.120 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:57 smithi158 ceph-mon[128925]: osdmap e65: 10 total, 10 up, 10 in 2023-07-20T02:49:58.120 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:57 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T02:49:58.120 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:57 smithi158 ceph-mon[128925]: Detected new or changed devices on smithi007 2023-07-20T02:49:58.121 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:57 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:58.121 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:57 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:58.121 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:57 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:49:58.121 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:57 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:49:58.122 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:57 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:49:58.122 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:57 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:49:58.122 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:57 smithi158 ceph-mon[128925]: Adjusting osd_memory_target on smithi007 to 4318M 2023-07-20T02:49:58.122 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:57 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:58.123 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:57 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:49:58.123 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:57 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:49:58.123 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:57 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:58.123 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:57 smithi158 ceph-mon[128925]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:49:58.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:57 smithi188 ceph-mon[129553]: purged_snaps scrub starts 2023-07-20T02:49:58.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:57 smithi188 ceph-mon[129553]: purged_snaps scrub ok 2023-07-20T02:49:58.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:57 smithi188 ceph-mon[129553]: osd.9 [v2:172.21.15.7:6826/1612464825,v1:172.21.15.7:6827/1612464825] boot 2023-07-20T02:49:58.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:57 smithi188 ceph-mon[129553]: osdmap e65: 10 total, 10 up, 10 in 2023-07-20T02:49:58.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:57 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T02:49:58.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:57 smithi188 ceph-mon[129553]: Detected new or changed devices on smithi007 2023-07-20T02:49:58.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:57 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:58.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:57 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:58.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:57 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:49:58.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:57 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:49:58.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:57 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:49:58.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:57 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:49:58.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:57 smithi188 ceph-mon[129553]: Adjusting osd_memory_target on smithi007 to 4318M 2023-07-20T02:49:58.215 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:57 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:58.215 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:57 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:49:58.215 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:57 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:49:58.215 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:57 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:58.215 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:57 smithi188 ceph-mon[129553]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:49:58.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.247+0000 7eff08e4a700 1 -- 172.21.15.158:0/2295490167 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff041012d0 msgr2=0x7eff04107b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:49:58.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.247+0000 7eff08e4a700 1 --2- 172.21.15.158:0/2295490167 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff041012d0 0x7eff04107b60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7efefc00b0a0 tx=0x7efefc00b3b0 comp rx=0 tx=0).stop 2023-07-20T02:49:58.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.247+0000 7eff08e4a700 1 -- 172.21.15.158:0/2295490167 shutdown_connections 2023-07-20T02:49:58.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.247+0000 7eff08e4a700 1 --2- 172.21.15.158:0/2295490167 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff041012d0 0x7eff04107b60 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:49:58.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.247+0000 7eff08e4a700 1 --2- 172.21.15.158:0/2295490167 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff040ffbf0 0x7eff04100010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:49:58.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.247+0000 7eff08e4a700 1 --2- 172.21.15.158:0/2295490167 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff04100910 0x7eff04100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:49:58.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.247+0000 7eff08e4a700 1 -- 172.21.15.158:0/2295490167 >> 172.21.15.158:0/2295490167 conn(0x7eff040fa3f0 msgr2=0x7eff040fc830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:49:58.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.247+0000 7eff08e4a700 1 -- 172.21.15.158:0/2295490167 shutdown_connections 2023-07-20T02:49:58.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.247+0000 7eff08e4a700 1 -- 172.21.15.158:0/2295490167 wait complete. 2023-07-20T02:49:58.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.248+0000 7eff08e4a700 1 Processor -- start 2023-07-20T02:49:58.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.248+0000 7eff08e4a700 1 -- start start 2023-07-20T02:49:58.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.248+0000 7eff08e4a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff040ffbf0 0x7eff040ff2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:49:58.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.248+0000 7eff08e4a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff04100910 0x7eff0419b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:49:58.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.248+0000 7eff08e4a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff041012d0 0x7eff0419d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:49:58.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.248+0000 7eff08e4a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0419e140 con 0x7eff040ffbf0 2023-07-20T02:49:58.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.248+0000 7eff08e4a700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0419e2b0 con 0x7eff041012d0 2023-07-20T02:49:58.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.248+0000 7eff08e4a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0419e470 con 0x7eff04100910 2023-07-20T02:49:58.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.248+0000 7eff02d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff041012d0 0x7eff0419d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:49:58.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.248+0000 7eff02d9d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff041012d0 0x7eff0419d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.158:58522/0 (socket says 172.21.15.158:58522) 2023-07-20T02:49:58.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.248+0000 7eff02d9d700 1 -- 172.21.15.158:0/1546950556 learned_addr learned my addr 172.21.15.158:0/1546950556 (peer_addr_for_me v2:172.21.15.158:0/0) 2023-07-20T02:49:58.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.248+0000 7eff01d9b700 1 --2- 172.21.15.158:0/1546950556 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff04100910 0x7eff0419b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:49:58.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.249+0000 7eff0259c700 1 --2- 172.21.15.158:0/1546950556 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff040ffbf0 0x7eff040ff2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:49:58.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.249+0000 7eff02d9d700 1 -- 172.21.15.158:0/1546950556 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff04100910 msgr2=0x7eff0419b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:49:58.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.249+0000 7eff02d9d700 1 --2- 172.21.15.158:0/1546950556 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff04100910 0x7eff0419b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:49:58.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.249+0000 7eff02d9d700 1 -- 172.21.15.158:0/1546950556 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff040ffbf0 msgr2=0x7eff040ff2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:49:58.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.249+0000 7eff02d9d700 1 --2- 172.21.15.158:0/1546950556 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff040ffbf0 0x7eff040ff2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:49:58.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.249+0000 7eff02d9d700 1 -- 172.21.15.158:0/1546950556 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efefc009d00 con 0x7eff041012d0 2023-07-20T02:49:58.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.249+0000 7eff02d9d700 1 --2- 172.21.15.158:0/1546950556 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff041012d0 0x7eff0419d9a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efefc00b830 tx=0x7efefc004c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:49:58.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.249+0000 7efef37fe700 1 -- 172.21.15.158:0/1546950556 <== mon.2 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efefc00e040 con 0x7eff041012d0 2023-07-20T02:49:58.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.249+0000 7eff08e4a700 1 -- 172.21.15.158:0/1546950556 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff0419e6f0 con 0x7eff041012d0 2023-07-20T02:49:58.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.249+0000 7efef37fe700 1 -- 172.21.15.158:0/1546950556 <== mon.2 v2:172.21.15.158:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efefc01db50 con 0x7eff041012d0 2023-07-20T02:49:58.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.249+0000 7efef37fe700 1 -- 172.21.15.158:0/1546950556 <== mon.2 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efefc01bbd0 con 0x7eff041012d0 2023-07-20T02:49:58.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.249+0000 7eff08e4a700 1 -- 172.21.15.158:0/1546950556 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff0419eb60 con 0x7eff041012d0 2023-07-20T02:49:58.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.250+0000 7eff08e4a700 1 -- 172.21.15.158:0/1546950556 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff04106600 con 0x7eff041012d0 2023-07-20T02:49:58.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.251+0000 7efef37fe700 1 -- 172.21.15.158:0/1546950556 <== mon.2 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7efefc01dcc0 con 0x7eff041012d0 2023-07-20T02:49:58.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.252+0000 7efef37fe700 1 --2- 172.21.15.158:0/1546950556 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7efeec0a11d0 0x7efeec0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:49:58.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.252+0000 7efef37fe700 1 -- 172.21.15.158:0/1546950556 <== mon.2 v2:172.21.15.158:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6261+0+0 (secure 0 0 0) 0x7efefc0ce2a0 con 0x7eff041012d0 2023-07-20T02:49:58.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.252+0000 7eff0259c700 1 --2- 172.21.15.158:0/1546950556 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7efeec0a11d0 0x7efeec0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:49:58.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.253+0000 7eff0259c700 1 --2- 172.21.15.158:0/1546950556 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7efeec0a11d0 0x7efeec0a3690 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efef8005950 tx=0x7efef80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:49:58.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.255+0000 7efef37fe700 1 -- 172.21.15.158:0/1546950556 <== mon.2 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7efefc0981e0 con 0x7eff041012d0 2023-07-20T02:49:58.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:57 smithi007 ceph-mon[126353]: purged_snaps scrub starts 2023-07-20T02:49:58.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:57 smithi007 ceph-mon[126353]: purged_snaps scrub ok 2023-07-20T02:49:58.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:57 smithi007 ceph-mon[126353]: osd.9 [v2:172.21.15.7:6826/1612464825,v1:172.21.15.7:6827/1612464825] boot 2023-07-20T02:49:58.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:57 smithi007 ceph-mon[126353]: osdmap e65: 10 total, 10 up, 10 in 2023-07-20T02:49:58.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:57 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T02:49:58.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:57 smithi007 ceph-mon[126353]: Detected new or changed devices on smithi007 2023-07-20T02:49:58.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:57 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:58.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:57 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:58.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:57 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:49:58.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:57 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:49:58.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:57 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:49:58.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:57 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:49:58.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:57 smithi007 ceph-mon[126353]: Adjusting osd_memory_target on smithi007 to 4318M 2023-07-20T02:49:58.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:57 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:58.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:57 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:49:58.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:57 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:49:58.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:57 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:49:58.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:57 smithi007 ceph-mon[126353]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:49:58.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:49:58.449+0000 7eff08e4a700 1 -- 172.21.15.158:0/1546950556 --> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi158:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7eff04060bb0 con 0x7efeec0a11d0 2023-07-20T02:49:59.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:58 smithi188 ceph-mon[129553]: osdmap e66: 10 total, 10 up, 10 in 2023-07-20T02:49:59.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:58 smithi188 ceph-mon[129553]: from='client.24380 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi158:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:49:59.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:58 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:49:59.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:58 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:49:59.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:58 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:49:59.226 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:58 smithi158 ceph-mon[128925]: osdmap e66: 10 total, 10 up, 10 in 2023-07-20T02:49:59.227 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:58 smithi158 ceph-mon[128925]: from='client.24380 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi158:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:49:59.227 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:58 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:49:59.228 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:58 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:49:59.228 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:58 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:49:59.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:58 smithi007 ceph-mon[126353]: osdmap e66: 10 total, 10 up, 10 in 2023-07-20T02:49:59.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:58 smithi007 ceph-mon[126353]: from='client.24380 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi158:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:49:59.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:58 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:49:59.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:58 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:49:59.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:58 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:50:00.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:49:59 smithi188 ceph-mon[129553]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:50:00.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:49:59 smithi158 ceph-mon[128925]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:50:00.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:49:59 smithi007 ceph-mon[126353]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:50:01.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:00 smithi188 ceph-mon[129553]: overall HEALTH_OK 2023-07-20T02:50:01.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:00 smithi188 ceph-mon[129553]: from='client.? 172.21.15.158:0/1173630738' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2651548-fe46-4d2d-b911-278036412ea5"}]: dispatch 2023-07-20T02:50:01.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:00 smithi188 ceph-mon[129553]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2651548-fe46-4d2d-b911-278036412ea5"}]: dispatch 2023-07-20T02:50:01.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:00 smithi188 ceph-mon[129553]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c2651548-fe46-4d2d-b911-278036412ea5"}]': finished 2023-07-20T02:50:01.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:00 smithi188 ceph-mon[129553]: osdmap e67: 11 total, 10 up, 11 in 2023-07-20T02:50:01.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:00 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T02:50:01.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:00 smithi158 ceph-mon[128925]: overall HEALTH_OK 2023-07-20T02:50:01.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:00 smithi158 ceph-mon[128925]: from='client.? 172.21.15.158:0/1173630738' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2651548-fe46-4d2d-b911-278036412ea5"}]: dispatch 2023-07-20T02:50:01.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:00 smithi158 ceph-mon[128925]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2651548-fe46-4d2d-b911-278036412ea5"}]: dispatch 2023-07-20T02:50:01.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:00 smithi158 ceph-mon[128925]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c2651548-fe46-4d2d-b911-278036412ea5"}]': finished 2023-07-20T02:50:01.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:00 smithi158 ceph-mon[128925]: osdmap e67: 11 total, 10 up, 11 in 2023-07-20T02:50:01.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:00 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T02:50:01.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:00 smithi007 ceph-mon[126353]: overall HEALTH_OK 2023-07-20T02:50:01.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:00 smithi007 ceph-mon[126353]: from='client.? 172.21.15.158:0/1173630738' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2651548-fe46-4d2d-b911-278036412ea5"}]: dispatch 2023-07-20T02:50:01.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:00 smithi007 ceph-mon[126353]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2651548-fe46-4d2d-b911-278036412ea5"}]: dispatch 2023-07-20T02:50:01.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:00 smithi007 ceph-mon[126353]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c2651548-fe46-4d2d-b911-278036412ea5"}]': finished 2023-07-20T02:50:01.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:00 smithi007 ceph-mon[126353]: osdmap e67: 11 total, 10 up, 11 in 2023-07-20T02:50:01.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:00 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T02:50:02.212 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:01 smithi188 ceph-mon[129553]: from='client.? 172.21.15.158:0/1935992139' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:50:02.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:01 smithi188 ceph-mon[129553]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:50:02.389 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:01 smithi158 ceph-mon[128925]: from='client.? 172.21.15.158:0/1935992139' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:50:02.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:01 smithi158 ceph-mon[128925]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:50:02.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:01 smithi007 ceph-mon[126353]: from='client.? 172.21.15.158:0/1935992139' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:50:02.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:01 smithi007 ceph-mon[126353]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:50:04.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:04 smithi158 ceph-mon[128925]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:50:04.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:04 smithi007 ceph-mon[126353]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:50:04.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:04 smithi188 ceph-mon[129553]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:50:06.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:06 smithi158 ceph-mon[128925]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:50:06.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:06 smithi007 ceph-mon[126353]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:50:07.071 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:06 smithi188 ceph-mon[129553]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:50:08.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:08 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-20T02:50:08.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:08 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:50:08.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:08 smithi158 ceph-mon[128925]: Deploying daemon osd.10 on smithi158 2023-07-20T02:50:08.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:08 smithi158 ceph-mon[128925]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:50:08.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:08 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-20T02:50:08.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:08 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:50:08.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:08 smithi007 ceph-mon[126353]: Deploying daemon osd.10 on smithi158 2023-07-20T02:50:08.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:08 smithi007 ceph-mon[126353]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:50:08.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:08 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-20T02:50:08.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:08 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:50:08.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:08 smithi188 ceph-mon[129553]: Deploying daemon osd.10 on smithi158 2023-07-20T02:50:08.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:08 smithi188 ceph-mon[129553]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:50:10.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:10 smithi158 ceph-mon[128925]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:50:10.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:10 smithi007 ceph-mon[126353]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:50:10.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:10 smithi188 ceph-mon[129553]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:50:12.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:12 smithi158 ceph-mon[128925]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:50:12.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:12 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:12.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:12 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:12.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:12 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:50:12.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:12 smithi007 ceph-mon[126353]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:50:12.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:12 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:12.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:12 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:12.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:12 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:50:12.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:12 smithi188 ceph-mon[129553]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:50:12.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:12 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:12.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:12 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:12.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:12 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:50:14.329 INFO:teuthology.orchestra.run.smithi158.stdout:Created osd(s) 10 on host 'smithi158' 2023-07-20T02:50:14.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:50:14.316+0000 7efef37fe700 1 -- 172.21.15.158:0/1546950556 <== mgr.14152 v2:172.21.15.7:6800/1354542137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7eff04060bb0 con 0x7efeec0a11d0 2023-07-20T02:50:14.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:50:14.319+0000 7eff08e4a700 1 -- 172.21.15.158:0/1546950556 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7efeec0a11d0 msgr2=0x7efeec0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:14.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:50:14.319+0000 7eff08e4a700 1 --2- 172.21.15.158:0/1546950556 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7efeec0a11d0 0x7efeec0a3690 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efef8005950 tx=0x7efef80058e0 comp rx=0 tx=0).stop 2023-07-20T02:50:14.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:50:14.319+0000 7eff08e4a700 1 -- 172.21.15.158:0/1546950556 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff041012d0 msgr2=0x7eff0419d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:14.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:50:14.319+0000 7eff08e4a700 1 --2- 172.21.15.158:0/1546950556 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff041012d0 0x7eff0419d9a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efefc00b830 tx=0x7efefc004c80 comp rx=0 tx=0).stop 2023-07-20T02:50:14.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:50:14.319+0000 7eff08e4a700 1 -- 172.21.15.158:0/1546950556 shutdown_connections 2023-07-20T02:50:14.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:50:14.319+0000 7eff08e4a700 1 --2- 172.21.15.158:0/1546950556 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7eff041012d0 0x7eff0419d9a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:14.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:50:14.319+0000 7eff08e4a700 1 --2- 172.21.15.158:0/1546950556 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7efeec0a11d0 0x7efeec0a3690 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:14.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:50:14.319+0000 7eff08e4a700 1 --2- 172.21.15.158:0/1546950556 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff040ffbf0 0x7eff040ff2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:14.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:50:14.319+0000 7eff08e4a700 1 --2- 172.21.15.158:0/1546950556 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff04100910 0x7eff0419b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:14.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:50:14.319+0000 7eff08e4a700 1 -- 172.21.15.158:0/1546950556 >> 172.21.15.158:0/1546950556 conn(0x7eff040fa3f0 msgr2=0x7eff040fc620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:50:14.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:50:14.319+0000 7eff08e4a700 1 -- 172.21.15.158:0/1546950556 shutdown_connections 2023-07-20T02:50:14.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-07-20T02:50:14.319+0000 7eff08e4a700 1 -- 172.21.15.158:0/1546950556 wait complete. 2023-07-20T02:50:14.770 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:14 smithi158 ceph-mon[128925]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:50:14.771 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:14 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:14.771 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:14 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:14.772 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:14 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:50:14.772 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:14 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:50:14.772 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:14 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:14.773 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:14 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:14.773 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:14 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:14.773 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:14 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:50:14.775 DEBUG:teuthology.orchestra.run.smithi158:osd.10> sudo journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.10.service 2023-07-20T02:50:14.778 INFO:tasks.cephadm:Deploying osd.11 on smithi188 with /dev/vg_nvme/lv_1... 2023-07-20T02:50:14.778 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-20T02:50:14.809 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:14 smithi188 ceph-mon[129553]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:50:14.809 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:14 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:14.809 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:14 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:14.809 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:14 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:50:14.809 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:14 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:50:14.810 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:14 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:14.810 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:14 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:14.810 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:14 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:14.810 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:14 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:50:14.816 INFO:journalctl@ceph.osd.10.smithi158.stdout:-- Logs begin at Thu 2023-07-20 02:10:12 UTC. -- 2023-07-20T02:50:14.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:14 smithi007 ceph-mon[126353]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:50:14.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:14 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:14.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:14 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:14.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:14 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:50:14.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:14 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:50:14.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:14 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:14.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:14 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:14.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:14 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:14.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:14 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:50:15.345 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T02:50:15.545 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:50:15 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:50:15.499+0000 7fddc2330640 -1 osd.10 0 log_to_monitors true 2023-07-20T02:50:15.797 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:15 smithi158 ceph-mon[128925]: from='osd.10 [v2:172.21.15.158:6824/1653166210,v1:172.21.15.158:6825/1653166210]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T02:50:15.797 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:15 smithi158 ceph-mon[128925]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T02:50:15.864 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:15 smithi188 ceph-mon[129553]: from='osd.10 [v2:172.21.15.158:6824/1653166210,v1:172.21.15.158:6825/1653166210]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T02:50:15.864 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:15 smithi188 ceph-mon[129553]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T02:50:15.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:15 smithi007 ceph-mon[126353]: from='osd.10 [v2:172.21.15.158:6824/1653166210,v1:172.21.15.158:6825/1653166210]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T02:50:15.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:15 smithi007 ceph-mon[126353]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T02:50:16.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:16 smithi188 ceph-mon[129553]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:50:16.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:16 smithi188 ceph-mon[129553]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-20T02:50:16.714 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:16 smithi188 ceph-mon[129553]: osdmap e68: 11 total, 10 up, 11 in 2023-07-20T02:50:16.714 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:16 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T02:50:16.714 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:16 smithi188 ceph-mon[129553]: from='osd.10 [v2:172.21.15.158:6824/1653166210,v1:172.21.15.158:6825/1653166210]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]: dispatch 2023-07-20T02:50:16.714 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:16 smithi188 ceph-mon[129553]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]: dispatch 2023-07-20T02:50:16.853 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-20T02:50:16.854 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-20T02:50:16.854 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2023-07-20T02:50:16.854 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2023-07-20T02:50:16.854 INFO:teuthology.orchestra.run.smithi188.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0110695 s, 947 MB/s 2023-07-20T02:50:16.854 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 2023-07-20T02:50:16.854 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2023-07-20T02:50:16.890 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:50:16 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:50:16.547+0000 7fddb79a8700 -1 osd.10 0 waiting for initial osdmap 2023-07-20T02:50:16.890 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:50:16 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:50:16.560+0000 7fddb279c700 -1 osd.10 69 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T02:50:16.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:16 smithi158 ceph-mon[128925]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:50:16.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:16 smithi158 ceph-mon[128925]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-20T02:50:16.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:16 smithi158 ceph-mon[128925]: osdmap e68: 11 total, 10 up, 11 in 2023-07-20T02:50:16.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:16 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T02:50:16.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:16 smithi158 ceph-mon[128925]: from='osd.10 [v2:172.21.15.158:6824/1653166210,v1:172.21.15.158:6825/1653166210]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]: dispatch 2023-07-20T02:50:16.892 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:16 smithi158 ceph-mon[128925]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]: dispatch 2023-07-20T02:50:16.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:16 smithi007 ceph-mon[126353]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:50:16.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:16 smithi007 ceph-mon[126353]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-20T02:50:16.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:16 smithi007 ceph-mon[126353]: osdmap e68: 11 total, 10 up, 11 in 2023-07-20T02:50:16.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:16 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T02:50:16.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:16 smithi007 ceph-mon[126353]: from='osd.10 [v2:172.21.15.158:6824/1653166210,v1:172.21.15.158:6825/1653166210]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]: dispatch 2023-07-20T02:50:16.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:16 smithi007 ceph-mon[126353]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]: dispatch 2023-07-20T02:50:17.299 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi188:vg_nvme/lv_1 2023-07-20T02:50:17.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:17 smithi158 ceph-mon[128925]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]': finished 2023-07-20T02:50:17.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:17 smithi158 ceph-mon[128925]: osdmap e69: 11 total, 10 up, 11 in 2023-07-20T02:50:17.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:17 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T02:50:17.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:17 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T02:50:17.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:17 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:17.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:17 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:17.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:17 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:50:17.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:17 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:50:17.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:17 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:50:17.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:17 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:50:17.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:17 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:17.892 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:17 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:50:17.892 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:17 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:50:17.892 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:17 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:17.892 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:17 smithi158 ceph-mon[128925]: osd.10 [v2:172.21.15.158:6824/1653166210,v1:172.21.15.158:6825/1653166210] boot 2023-07-20T02:50:17.892 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:17 smithi158 ceph-mon[128925]: osdmap e70: 11 total, 11 up, 11 in 2023-07-20T02:50:17.892 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:17 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T02:50:17.893 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T02:50:17.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:17 smithi007 ceph-mon[126353]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]': finished 2023-07-20T02:50:17.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:17 smithi007 ceph-mon[126353]: osdmap e69: 11 total, 10 up, 11 in 2023-07-20T02:50:17.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:17 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T02:50:17.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:17 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T02:50:17.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:17 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:17.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:17 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:17.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:17 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:50:17.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:17 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:50:17.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:17 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:50:17.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:17 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:50:17.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:17 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:17.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:17 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:50:17.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:17 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:50:17.950 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:17 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:17.950 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:17 smithi007 ceph-mon[126353]: osd.10 [v2:172.21.15.158:6824/1653166210,v1:172.21.15.158:6825/1653166210] boot 2023-07-20T02:50:17.950 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:17 smithi007 ceph-mon[126353]: osdmap e70: 11 total, 11 up, 11 in 2023-07-20T02:50:17.950 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:17 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T02:50:17.962 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:17 smithi188 ceph-mon[129553]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]': finished 2023-07-20T02:50:17.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:17 smithi188 ceph-mon[129553]: osdmap e69: 11 total, 10 up, 11 in 2023-07-20T02:50:17.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:17 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T02:50:17.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:17 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T02:50:17.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:17 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:17.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:17 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:17.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:17 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:50:17.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:17 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:50:17.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:17 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:50:17.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:17 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:50:17.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:17 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:17.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:17 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:50:17.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:17 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:50:17.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:17 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:17.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:17 smithi188 ceph-mon[129553]: osd.10 [v2:172.21.15.158:6824/1653166210,v1:172.21.15.158:6825/1653166210] boot 2023-07-20T02:50:17.965 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:17 smithi188 ceph-mon[129553]: osdmap e70: 11 total, 11 up, 11 in 2023-07-20T02:50:17.965 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:17 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T02:50:18.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:18 smithi188 ceph-mon[129553]: purged_snaps scrub starts 2023-07-20T02:50:18.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:18 smithi188 ceph-mon[129553]: purged_snaps scrub ok 2023-07-20T02:50:18.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:18 smithi188 ceph-mon[129553]: Detected new or changed devices on smithi158 2023-07-20T02:50:18.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:18 smithi188 ceph-mon[129553]: Adjusting osd_memory_target on smithi158 to 4292M 2023-07-20T02:50:18.714 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:18 smithi188 ceph-mon[129553]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:50:18.714 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:18 smithi188 ceph-mon[129553]: osdmap e71: 11 total, 11 up, 11 in 2023-07-20T02:50:18.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:18 smithi158 ceph-mon[128925]: purged_snaps scrub starts 2023-07-20T02:50:18.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:18 smithi158 ceph-mon[128925]: purged_snaps scrub ok 2023-07-20T02:50:18.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:18 smithi158 ceph-mon[128925]: Detected new or changed devices on smithi158 2023-07-20T02:50:18.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:18 smithi158 ceph-mon[128925]: Adjusting osd_memory_target on smithi158 to 4292M 2023-07-20T02:50:18.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:18 smithi158 ceph-mon[128925]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:50:18.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:18 smithi158 ceph-mon[128925]: osdmap e71: 11 total, 11 up, 11 in 2023-07-20T02:50:18.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:18 smithi007 ceph-mon[126353]: purged_snaps scrub starts 2023-07-20T02:50:18.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:18 smithi007 ceph-mon[126353]: purged_snaps scrub ok 2023-07-20T02:50:18.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:18 smithi007 ceph-mon[126353]: Detected new or changed devices on smithi158 2023-07-20T02:50:18.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:18 smithi007 ceph-mon[126353]: Adjusting osd_memory_target on smithi158 to 4292M 2023-07-20T02:50:18.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:18 smithi007 ceph-mon[126353]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-20T02:50:18.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:18 smithi007 ceph-mon[126353]: osdmap e71: 11 total, 11 up, 11 in 2023-07-20T02:50:19.214 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.201+0000 7f17d7e80700 1 -- 172.21.15.188:0/4149033197 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17d0100910 msgr2=0x7f17d0100d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:19.215 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.201+0000 7f17d7e80700 1 --2- 172.21.15.188:0/4149033197 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17d0100910 0x7f17d0100d70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f17cc009990 tx=0x7f17cc009ca0 comp rx=0 tx=0).stop 2023-07-20T02:50:19.215 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.201+0000 7f17d7e80700 1 -- 172.21.15.188:0/4149033197 shutdown_connections 2023-07-20T02:50:19.215 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.201+0000 7f17d7e80700 1 --2- 172.21.15.188:0/4149033197 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17d01012b0 0x7f17d0107b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:19.215 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.201+0000 7f17d7e80700 1 --2- 172.21.15.188:0/4149033197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17d00ff710 0x7f17d00ffb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:19.215 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.201+0000 7f17d7e80700 1 --2- 172.21.15.188:0/4149033197 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17d0100910 0x7f17d0100d70 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:19.215 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.201+0000 7f17d7e80700 1 -- 172.21.15.188:0/4149033197 >> 172.21.15.188:0/4149033197 conn(0x7f17d00facb0 msgr2=0x7f17d00fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:50:19.216 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.201+0000 7f17d7e80700 1 -- 172.21.15.188:0/4149033197 shutdown_connections 2023-07-20T02:50:19.216 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.201+0000 7f17d7e80700 1 -- 172.21.15.188:0/4149033197 wait complete. 2023-07-20T02:50:19.217 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.202+0000 7f17d7e80700 1 Processor -- start 2023-07-20T02:50:19.217 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.202+0000 7f17d7e80700 1 -- start start 2023-07-20T02:50:19.217 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.202+0000 7f17d7e80700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17d00ff710 0x7f17d0196490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:19.218 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.202+0000 7f17d7e80700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17d0100910 0x7f17d01969d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:19.218 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.202+0000 7f17d7e80700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17d01012b0 0x7f17d019da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:19.218 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.202+0000 7f17d7e80700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d019e020 con 0x7f17d00ff710 2023-07-20T02:50:19.218 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.202+0000 7f17d7e80700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d019e160 con 0x7f17d0100910 2023-07-20T02:50:19.218 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.202+0000 7f17d7e80700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d019e2f0 con 0x7f17d01012b0 2023-07-20T02:50:19.218 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.202+0000 7f17d641d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17d01012b0 0x7f17d019da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:19.218 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.202+0000 7f17d641d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17d01012b0 0x7f17d019da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:35212/0 (socket says 172.21.15.188:35212) 2023-07-20T02:50:19.219 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.202+0000 7f17d641d700 1 -- 172.21.15.188:0/1228857130 learned_addr learned my addr 172.21.15.188:0/1228857130 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-07-20T02:50:19.219 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.202+0000 7f17d541b700 1 --2- 172.21.15.188:0/1228857130 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17d0100910 0x7f17d01969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:19.219 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.202+0000 7f17d5c1c700 1 --2- 172.21.15.188:0/1228857130 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17d00ff710 0x7f17d0196490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:19.219 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.203+0000 7f17d641d700 1 -- 172.21.15.188:0/1228857130 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17d0100910 msgr2=0x7f17d01969d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:19.219 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.203+0000 7f17d641d700 1 --2- 172.21.15.188:0/1228857130 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17d0100910 0x7f17d01969d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:19.219 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.203+0000 7f17d641d700 1 -- 172.21.15.188:0/1228857130 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17d00ff710 msgr2=0x7f17d0196490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:19.219 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.203+0000 7f17d641d700 1 --2- 172.21.15.188:0/1228857130 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17d00ff710 0x7f17d0196490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:19.219 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.203+0000 7f17d641d700 1 -- 172.21.15.188:0/1228857130 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17cc0095f0 con 0x7f17d01012b0 2023-07-20T02:50:19.220 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.203+0000 7f17d641d700 1 --2- 172.21.15.188:0/1228857130 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17d01012b0 0x7f17d019da50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f17c8007f10 tx=0x7f17c800d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:50:19.220 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.203+0000 7f17c6ffd700 1 -- 172.21.15.188:0/1228857130 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c8017070 con 0x7f17d01012b0 2023-07-20T02:50:19.220 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.203+0000 7f17c6ffd700 1 -- 172.21.15.188:0/1228857130 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f17c800f040 con 0x7f17d01012b0 2023-07-20T02:50:19.220 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.203+0000 7f17d7e80700 1 -- 172.21.15.188:0/1228857130 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17d019e5d0 con 0x7f17d01012b0 2023-07-20T02:50:19.220 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.203+0000 7f17c6ffd700 1 -- 172.21.15.188:0/1228857130 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c8005610 con 0x7f17d01012b0 2023-07-20T02:50:19.220 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.203+0000 7f17d7e80700 1 -- 172.21.15.188:0/1228857130 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17d019ea80 con 0x7f17d01012b0 2023-07-20T02:50:19.220 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.204+0000 7f17d7e80700 1 -- 172.21.15.188:0/1228857130 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17d010af20 con 0x7f17d01012b0 2023-07-20T02:50:19.221 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.206+0000 7f17c6ffd700 1 -- 172.21.15.188:0/1228857130 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f17c8004ad0 con 0x7f17d01012b0 2023-07-20T02:50:19.221 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.206+0000 7f17c6ffd700 1 --2- 172.21.15.188:0/1228857130 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f17bc0a1150 0x7f17bc0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:19.221 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.206+0000 7f17c6ffd700 1 -- 172.21.15.188:0/1228857130 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6723+0+0 (secure 0 0 0) 0x7f17c80cc0a0 con 0x7f17d01012b0 2023-07-20T02:50:19.222 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.207+0000 7f17d5c1c700 1 --2- 172.21.15.188:0/1228857130 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f17bc0a1150 0x7f17bc0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:19.222 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.207+0000 7f17d5c1c700 1 --2- 172.21.15.188:0/1228857130 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f17bc0a1150 0x7f17bc0a3610 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f17d0100770 tx=0x7f17c000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:50:19.222 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.210+0000 7f17c6ffd700 1 -- 172.21.15.188:0/1228857130 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f17c8095e20 con 0x7f17d01012b0 2023-07-20T02:50:19.416 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:19.403+0000 7f17d7e80700 1 -- 172.21.15.188:0/1228857130 --> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f17d00692e0 con 0x7f17bc0a1150 2023-07-20T02:50:20.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:20 smithi158 ceph-mon[128925]: osdmap e72: 11 total, 11 up, 11 in 2023-07-20T02:50:20.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:20 smithi158 ceph-mon[128925]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:50:20.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:20 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:50:20.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:20 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:50:20.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:20 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:50:20.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:20 smithi158 ceph-mon[128925]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:50:20.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:20 smithi007 ceph-mon[126353]: osdmap e72: 11 total, 11 up, 11 in 2023-07-20T02:50:20.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:20 smithi007 ceph-mon[126353]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:50:20.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:20 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:50:20.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:20 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:50:20.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:20 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:50:20.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:20 smithi007 ceph-mon[126353]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:50:20.454 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:20 smithi188 ceph-mon[129553]: osdmap e72: 11 total, 11 up, 11 in 2023-07-20T02:50:20.455 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:20 smithi188 ceph-mon[129553]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:50:20.455 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:20 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T02:50:20.455 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:20 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T02:50:20.455 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:20 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:50:20.455 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:20 smithi188 ceph-mon[129553]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:50:21.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:21 smithi007 ceph-mon[126353]: osdmap e73: 11 total, 11 up, 11 in 2023-07-20T02:50:21.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:21 smithi188 ceph-mon[129553]: osdmap e73: 11 total, 11 up, 11 in 2023-07-20T02:50:21.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:21 smithi158 ceph-mon[128925]: osdmap e73: 11 total, 11 up, 11 in 2023-07-20T02:50:22.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:22 smithi007 ceph-mon[126353]: osdmap e74: 11 total, 11 up, 11 in 2023-07-20T02:50:22.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:22 smithi007 ceph-mon[126353]: pgmap v228: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-20T02:50:22.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:22 smithi007 ceph-mon[126353]: from='client.? 172.21.15.188:0/1298413982' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7262bcb-4243-403a-a321-fc2bdf1d4d25"}]: dispatch 2023-07-20T02:50:22.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:22 smithi007 ceph-mon[126353]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7262bcb-4243-403a-a321-fc2bdf1d4d25"}]: dispatch 2023-07-20T02:50:22.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:22 smithi007 ceph-mon[126353]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7262bcb-4243-403a-a321-fc2bdf1d4d25"}]': finished 2023-07-20T02:50:22.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:22 smithi007 ceph-mon[126353]: osdmap e75: 12 total, 11 up, 12 in 2023-07-20T02:50:22.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:22 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T02:50:22.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:22 smithi188 ceph-mon[129553]: osdmap e74: 11 total, 11 up, 11 in 2023-07-20T02:50:22.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:22 smithi188 ceph-mon[129553]: pgmap v228: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-20T02:50:22.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:22 smithi188 ceph-mon[129553]: from='client.? 172.21.15.188:0/1298413982' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7262bcb-4243-403a-a321-fc2bdf1d4d25"}]: dispatch 2023-07-20T02:50:22.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:22 smithi188 ceph-mon[129553]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7262bcb-4243-403a-a321-fc2bdf1d4d25"}]: dispatch 2023-07-20T02:50:22.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:22 smithi188 ceph-mon[129553]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7262bcb-4243-403a-a321-fc2bdf1d4d25"}]': finished 2023-07-20T02:50:22.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:22 smithi188 ceph-mon[129553]: osdmap e75: 12 total, 11 up, 12 in 2023-07-20T02:50:22.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:22 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T02:50:22.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:22 smithi158 ceph-mon[128925]: osdmap e74: 11 total, 11 up, 11 in 2023-07-20T02:50:22.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:22 smithi158 ceph-mon[128925]: pgmap v228: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-20T02:50:22.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:22 smithi158 ceph-mon[128925]: from='client.? 172.21.15.188:0/1298413982' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7262bcb-4243-403a-a321-fc2bdf1d4d25"}]: dispatch 2023-07-20T02:50:22.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:22 smithi158 ceph-mon[128925]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7262bcb-4243-403a-a321-fc2bdf1d4d25"}]: dispatch 2023-07-20T02:50:22.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:22 smithi158 ceph-mon[128925]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7262bcb-4243-403a-a321-fc2bdf1d4d25"}]': finished 2023-07-20T02:50:22.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:22 smithi158 ceph-mon[128925]: osdmap e75: 12 total, 11 up, 12 in 2023-07-20T02:50:22.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:22 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T02:50:23.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:23 smithi007 ceph-mon[126353]: from='client.? 172.21.15.188:0/2968607104' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:50:23.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:23 smithi188 ceph-mon[129553]: from='client.? 172.21.15.188:0/2968607104' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:50:23.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:23 smithi158 ceph-mon[128925]: from='client.? 172.21.15.188:0/2968607104' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T02:50:24.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:24 smithi007 ceph-mon[126353]: pgmap v230: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail; 131 KiB/s, 0 objects/s recovering 2023-07-20T02:50:24.462 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:24 smithi188 ceph-mon[129553]: pgmap v230: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail; 131 KiB/s, 0 objects/s recovering 2023-07-20T02:50:24.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:24 smithi158 ceph-mon[128925]: pgmap v230: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail; 131 KiB/s, 0 objects/s recovering 2023-07-20T02:50:26.885 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:26 smithi188 ceph-mon[129553]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T02:50:26.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:26 smithi158 ceph-mon[128925]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T02:50:26.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:26 smithi007 ceph-mon[126353]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T02:50:28.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:28 smithi158 ceph-mon[128925]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:50:28.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:28 smithi007 ceph-mon[126353]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:50:28.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:28 smithi188 ceph-mon[129553]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:50:29.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:29 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-20T02:50:29.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:29 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:50:29.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:29 smithi158 ceph-mon[128925]: Deploying daemon osd.11 on smithi188 2023-07-20T02:50:29.427 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:29 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-20T02:50:29.428 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:29 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:50:29.428 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:29 smithi188 ceph-mon[129553]: Deploying daemon osd.11 on smithi188 2023-07-20T02:50:29.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:29 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-20T02:50:29.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:29 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:50:29.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:29 smithi007 ceph-mon[126353]: Deploying daemon osd.11 on smithi188 2023-07-20T02:50:30.370 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:30 smithi188 ceph-mon[129553]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:50:30.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:30 smithi158 ceph-mon[128925]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:50:30.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:30 smithi007 ceph-mon[126353]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:50:32.853 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:32 smithi188 ceph-mon[129553]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:50:32.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:32 smithi158 ceph-mon[128925]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:50:32.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:32 smithi007 ceph-mon[126353]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:50:34.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:34 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:34.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:34 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:34.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:34 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:50:34.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:34 smithi188 ceph-mon[129553]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:50:34.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:34 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:34.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:34 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:34.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:34 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:50:34.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:34 smithi158 ceph-mon[128925]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:50:34.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:34 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:34.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:34 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:34.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:34 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:50:34.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:34 smithi007 ceph-mon[126353]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:50:36.290 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 11 on host 'smithi188' 2023-07-20T02:50:36.291 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:36.278+0000 7f17c6ffd700 1 -- 172.21.15.188:0/1228857130 <== mgr.14152 v2:172.21.15.7:6800/1354542137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f17d00692e0 con 0x7f17bc0a1150 2023-07-20T02:50:36.293 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:36.281+0000 7f17d7e80700 1 -- 172.21.15.188:0/1228857130 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f17bc0a1150 msgr2=0x7f17bc0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:36.293 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:36.281+0000 7f17d7e80700 1 --2- 172.21.15.188:0/1228857130 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f17bc0a1150 0x7f17bc0a3610 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f17d0100770 tx=0x7f17c000a380 comp rx=0 tx=0).stop 2023-07-20T02:50:36.293 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:36.281+0000 7f17d7e80700 1 -- 172.21.15.188:0/1228857130 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17d01012b0 msgr2=0x7f17d019da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:36.293 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:36.281+0000 7f17d7e80700 1 --2- 172.21.15.188:0/1228857130 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17d01012b0 0x7f17d019da50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f17c8007f10 tx=0x7f17c800d3b0 comp rx=0 tx=0).stop 2023-07-20T02:50:36.294 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:36.281+0000 7f17d7e80700 1 -- 172.21.15.188:0/1228857130 shutdown_connections 2023-07-20T02:50:36.294 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:36.281+0000 7f17d7e80700 1 --2- 172.21.15.188:0/1228857130 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17d01012b0 0x7f17d019da50 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:36.294 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:36.281+0000 7f17d7e80700 1 --2- 172.21.15.188:0/1228857130 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f17bc0a1150 0x7f17bc0a3610 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:36.294 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:36.281+0000 7f17d7e80700 1 --2- 172.21.15.188:0/1228857130 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17d00ff710 0x7f17d0196490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:36.294 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:36.281+0000 7f17d7e80700 1 --2- 172.21.15.188:0/1228857130 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f17d0100910 0x7f17d01969d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:36.295 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:36.281+0000 7f17d7e80700 1 -- 172.21.15.188:0/1228857130 >> 172.21.15.188:0/1228857130 conn(0x7f17d00facb0 msgr2=0x7f17d0068bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:50:36.295 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:36.281+0000 7f17d7e80700 1 -- 172.21.15.188:0/1228857130 shutdown_connections 2023-07-20T02:50:36.295 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:36.281+0000 7f17d7e80700 1 -- 172.21.15.188:0/1228857130 wait complete. 2023-07-20T02:50:36.784 DEBUG:teuthology.orchestra.run.smithi188:osd.11> sudo journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.11.service 2023-07-20T02:50:36.787 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2023-07-20T02:50:36.787 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph osd stat -f json 2023-07-20T02:50:36.810 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:36 smithi188 ceph-mon[129553]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:50:36.810 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:36 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:36.811 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:36 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:36.811 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:36 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:50:36.811 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:36 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:50:36.811 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:36 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:36.811 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:36 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:36.811 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:36 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:36.811 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:36 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:50:36.816 INFO:journalctl@ceph.osd.11.smithi188.stdout:-- Logs begin at Thu 2023-07-20 02:09:43 UTC. -- 2023-07-20T02:50:36.858 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:36 smithi007 ceph-mon[126353]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:50:36.859 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:36 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:36.859 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:36 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:36.859 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:36 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:50:36.859 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:36 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:50:36.859 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:36 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:36.859 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:36 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:36.860 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:36 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:36.860 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:36 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:50:36.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:36 smithi158 ceph-mon[128925]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:50:36.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:36 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:36.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:36 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:36.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:36 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:50:36.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:36 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:50:36.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:36 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:36.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:36 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:36.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:36 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:36.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:36 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:50:37.463 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:50:37 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:50:37.179+0000 7f29fb054640 -1 osd.11 0 log_to_monitors true 2023-07-20T02:50:37.533 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:50:38.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:38 smithi188 ceph-mon[129553]: from='osd.11 [v2:172.21.15.188:6824/1230196525,v1:172.21.15.188:6825/1230196525]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T02:50:38.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:38 smithi188 ceph-mon[129553]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T02:50:38.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:38 smithi188 ceph-mon[129553]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:50:38.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:38 smithi158 ceph-mon[128925]: from='osd.11 [v2:172.21.15.188:6824/1230196525,v1:172.21.15.188:6825/1230196525]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T02:50:38.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:38 smithi158 ceph-mon[128925]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T02:50:38.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:38 smithi158 ceph-mon[128925]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:50:38.409 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:38 smithi007 ceph-mon[126353]: from='osd.11 [v2:172.21.15.188:6824/1230196525,v1:172.21.15.188:6825/1230196525]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T02:50:38.409 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:38 smithi007 ceph-mon[126353]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T02:50:38.409 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:38 smithi007 ceph-mon[126353]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:50:39.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.988+0000 7f67024a9700 1 -- 172.21.15.7:0/2670853193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66fc101a00 msgr2=0x7f66fc101e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:39.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.988+0000 7f67024a9700 1 --2- 172.21.15.7:0/2670853193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66fc101a00 0x7f66fc101e60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f66f00099c0 tx=0x7f66f0009cd0 comp rx=0 tx=0).stop 2023-07-20T02:50:39.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.988+0000 7f67024a9700 1 -- 172.21.15.7:0/2670853193 shutdown_connections 2023-07-20T02:50:39.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.988+0000 7f67024a9700 1 --2- 172.21.15.7:0/2670853193 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f66fc1023a0 0x7f66fc108c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:39.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.988+0000 7f67024a9700 1 --2- 172.21.15.7:0/2670853193 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f66fc100800 0x7f66fc100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:39.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.988+0000 7f67024a9700 1 --2- 172.21.15.7:0/2670853193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66fc101a00 0x7f66fc101e60 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:39.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.988+0000 7f67024a9700 1 -- 172.21.15.7:0/2670853193 >> 172.21.15.7:0/2670853193 conn(0x7f66fc0fbd60 msgr2=0x7f66fc0fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:50:39.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.988+0000 7f67024a9700 1 -- 172.21.15.7:0/2670853193 shutdown_connections 2023-07-20T02:50:39.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.988+0000 7f67024a9700 1 -- 172.21.15.7:0/2670853193 wait complete. 2023-07-20T02:50:39.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.989+0000 7f67024a9700 1 Processor -- start 2023-07-20T02:50:39.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.990+0000 7f67024a9700 1 -- start start 2023-07-20T02:50:39.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.990+0000 7f67024a9700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f66fc100800 0x7f66fc196530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:39.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.990+0000 7f67024a9700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f66fc101a00 0x7f66fc196a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:39.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.990+0000 7f67024a9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66fc1023a0 0x7f66fc19daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:39.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.990+0000 7f67024a9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc19e0c0 con 0x7f66fc1023a0 2023-07-20T02:50:39.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.990+0000 7f67024a9700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc19e200 con 0x7f66fc101a00 2023-07-20T02:50:39.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.990+0000 7f67024a9700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc19e390 con 0x7f66fc100800 2023-07-20T02:50:39.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.990+0000 7f66fbfff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f66fc100800 0x7f66fc196530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:39.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.990+0000 7f66fbfff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f66fc100800 0x7f66fc196530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.7:57618/0 (socket says 172.21.15.7:57618) 2023-07-20T02:50:39.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.990+0000 7f66fbfff700 1 -- 172.21.15.7:0/4077569660 learned_addr learned my addr 172.21.15.7:0/4077569660 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:50:39.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.990+0000 7f6700a46700 1 --2- 172.21.15.7:0/4077569660 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66fc1023a0 0x7f66fc19daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:39.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.990+0000 7f66fb7fe700 1 --2- 172.21.15.7:0/4077569660 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f66fc101a00 0x7f66fc196a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:39.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.991+0000 7f66fbfff700 1 -- 172.21.15.7:0/4077569660 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f66fc101a00 msgr2=0x7f66fc196a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:39.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.991+0000 7f66fbfff700 1 --2- 172.21.15.7:0/4077569660 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f66fc101a00 0x7f66fc196a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:39.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.991+0000 7f66fbfff700 1 -- 172.21.15.7:0/4077569660 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66fc1023a0 msgr2=0x7f66fc19daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:39.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.991+0000 7f66fbfff700 1 --2- 172.21.15.7:0/4077569660 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66fc1023a0 0x7f66fc19daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:39.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.991+0000 7f66fbfff700 1 -- 172.21.15.7:0/4077569660 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ec009520 con 0x7f66fc100800 2023-07-20T02:50:39.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.991+0000 7f6700a46700 1 --2- 172.21.15.7:0/4077569660 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66fc1023a0 0x7f66fc19daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T02:50:39.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.991+0000 7f66fbfff700 1 --2- 172.21.15.7:0/4077569660 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f66fc100800 0x7f66fc196530 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f66ec00ea80 tx=0x7f66ec00ed90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:50:39.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.991+0000 7f66f97fa700 1 -- 172.21.15.7:0/4077569660 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66ec00cbd0 con 0x7f66fc100800 2023-07-20T02:50:39.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.991+0000 7f66f97fa700 1 -- 172.21.15.7:0/4077569660 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f66ec00cd30 con 0x7f66fc100800 2023-07-20T02:50:39.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.991+0000 7f67024a9700 1 -- 172.21.15.7:0/4077569660 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66f00095f0 con 0x7f66fc100800 2023-07-20T02:50:39.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.991+0000 7f66f97fa700 1 -- 172.21.15.7:0/4077569660 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66ec018680 con 0x7f66fc100800 2023-07-20T02:50:39.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.991+0000 7f67024a9700 1 -- 172.21.15.7:0/4077569660 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66fc19e9e0 con 0x7f66fc100800 2023-07-20T02:50:39.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.992+0000 7f67024a9700 1 -- 172.21.15.7:0/4077569660 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66fc10c050 con 0x7f66fc100800 2023-07-20T02:50:39.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.994+0000 7f66f97fa700 1 -- 172.21.15.7:0/4077569660 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f66ec0187e0 con 0x7f66fc100800 2023-07-20T02:50:39.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.996+0000 7f66f97fa700 1 --2- 172.21.15.7:0/4077569660 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f66e40a1150 0x7f66e40a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:39.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.996+0000 7f66f97fa700 1 -- 172.21.15.7:0/4077569660 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6831+0+0 (secure 0 0 0) 0x7f66ec014070 con 0x7f66fc100800 2023-07-20T02:50:39.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.996+0000 7f66fb7fe700 1 --2- 172.21.15.7:0/4077569660 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f66e40a1150 0x7f66e40a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:39.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:38.996+0000 7f66fb7fe700 1 --2- 172.21.15.7:0/4077569660 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f66e40a1150 0x7f66e40a3610 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f66f000b580 tx=0x7f66f0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:50:39.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:39.000+0000 7f66f97fa700 1 -- 172.21.15.7:0/4077569660 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f66ec0963e0 con 0x7f66fc100800 2023-07-20T02:50:39.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:39.213+0000 7f67024a9700 1 -- 172.21.15.7:0/4077569660 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f66fc10c240 con 0x7f66fc100800 2023-07-20T02:50:39.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:39.213+0000 7f66f97fa700 1 -- 172.21.15.7:0/4077569660 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v77) v1 ==== 74+0+133 (secure 0 0 0) 0x7f66ec099a00 con 0x7f66fc100800 2023-07-20T02:50:39.227 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:50:39.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:39.216+0000 7f67024a9700 1 -- 172.21.15.7:0/4077569660 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f66e40a1150 msgr2=0x7f66e40a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:39.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:39.216+0000 7f67024a9700 1 --2- 172.21.15.7:0/4077569660 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f66e40a1150 0x7f66e40a3610 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f66f000b580 tx=0x7f66f0005fb0 comp rx=0 tx=0).stop 2023-07-20T02:50:39.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:39.216+0000 7f67024a9700 1 -- 172.21.15.7:0/4077569660 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f66fc100800 msgr2=0x7f66fc196530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:39.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:39.216+0000 7f67024a9700 1 --2- 172.21.15.7:0/4077569660 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f66fc100800 0x7f66fc196530 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f66ec00ea80 tx=0x7f66ec00ed90 comp rx=0 tx=0).stop 2023-07-20T02:50:39.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:39.216+0000 7f67024a9700 1 -- 172.21.15.7:0/4077569660 shutdown_connections 2023-07-20T02:50:39.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:39.216+0000 7f67024a9700 1 --2- 172.21.15.7:0/4077569660 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f66e40a1150 0x7f66e40a3610 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:39.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:39.216+0000 7f67024a9700 1 --2- 172.21.15.7:0/4077569660 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66fc1023a0 0x7f66fc19daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:39.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:39.216+0000 7f67024a9700 1 --2- 172.21.15.7:0/4077569660 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f66fc100800 0x7f66fc196530 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:39.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:39.216+0000 7f67024a9700 1 --2- 172.21.15.7:0/4077569660 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f66fc101a00 0x7f66fc196a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:39.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:39.216+0000 7f67024a9700 1 -- 172.21.15.7:0/4077569660 >> 172.21.15.7:0/4077569660 conn(0x7f66fc0fbd60 msgr2=0x7f66fc0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:50:39.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:39.217+0000 7f67024a9700 1 -- 172.21.15.7:0/4077569660 shutdown_connections 2023-07-20T02:50:39.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:39.217+0000 7f67024a9700 1 -- 172.21.15.7:0/4077569660 wait complete. 2023-07-20T02:50:39.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:39 smithi158 ceph-mon[128925]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-20T02:50:39.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:39 smithi158 ceph-mon[128925]: osdmap e76: 12 total, 11 up, 12 in 2023-07-20T02:50:39.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:39 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T02:50:39.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:39 smithi158 ceph-mon[128925]: from='osd.11 [v2:172.21.15.188:6824/1230196525,v1:172.21.15.188:6825/1230196525]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-07-20T02:50:39.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:39 smithi158 ceph-mon[128925]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-07-20T02:50:39.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:39 smithi007 ceph-mon[126353]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-20T02:50:39.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:39 smithi007 ceph-mon[126353]: osdmap e76: 12 total, 11 up, 12 in 2023-07-20T02:50:39.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:39 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T02:50:39.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:39 smithi007 ceph-mon[126353]: from='osd.11 [v2:172.21.15.188:6824/1230196525,v1:172.21.15.188:6825/1230196525]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-07-20T02:50:39.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:39 smithi007 ceph-mon[126353]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-07-20T02:50:39.463 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:50:39 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:50:39.130+0000 7f29f06cc700 -1 osd.11 0 waiting for initial osdmap 2023-07-20T02:50:39.463 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:50:39 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:50:39.144+0000 7f29e8cbb700 -1 osd.11 77 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T02:50:39.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:39 smithi188 ceph-mon[129553]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-20T02:50:39.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:39 smithi188 ceph-mon[129553]: osdmap e76: 12 total, 11 up, 12 in 2023-07-20T02:50:39.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:39 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T02:50:39.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:39 smithi188 ceph-mon[129553]: from='osd.11 [v2:172.21.15.188:6824/1230196525,v1:172.21.15.188:6825/1230196525]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-07-20T02:50:39.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:39 smithi188 ceph-mon[129553]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2023-07-20T02:50:39.715 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":77,"num_osds":12,"num_up_osds":11,"osd_up_since":1689821417,"num_in_osds":12,"osd_in_since":1689821421,"num_remapped_pgs":0} 2023-07-20T02:50:40.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:40 smithi007 ceph-mon[126353]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2023-07-20T02:50:40.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:40 smithi007 ceph-mon[126353]: osdmap e77: 12 total, 11 up, 12 in 2023-07-20T02:50:40.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:40 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T02:50:40.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:40 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T02:50:40.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:40 smithi007 ceph-mon[126353]: Detected new or changed devices on smithi188 2023-07-20T02:50:40.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:40 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:40.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:40 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:40.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:40 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:50:40.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:40 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:50:40.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:40 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:50:40.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:40 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:50:40.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:40 smithi007 ceph-mon[126353]: Adjusting osd_memory_target on smithi188 to 4292M 2023-07-20T02:50:40.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:40 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:40.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:40 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:50:40.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:40 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/4077569660' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T02:50:40.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:40 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:50:40.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:40 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:40.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:40 smithi007 ceph-mon[126353]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:50:40.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:40 smithi188 ceph-mon[129553]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2023-07-20T02:50:40.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:40 smithi188 ceph-mon[129553]: osdmap e77: 12 total, 11 up, 12 in 2023-07-20T02:50:40.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:40 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T02:50:40.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:40 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T02:50:40.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:40 smithi188 ceph-mon[129553]: Detected new or changed devices on smithi188 2023-07-20T02:50:40.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:40 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:40.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:40 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:40.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:40 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:50:40.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:40 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:50:40.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:40 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:50:40.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:40 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:50:40.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:40 smithi188 ceph-mon[129553]: Adjusting osd_memory_target on smithi188 to 4292M 2023-07-20T02:50:40.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:40 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:40.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:40 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:50:40.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:40 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/4077569660' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T02:50:40.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:40 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:50:40.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:40 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:40.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:40 smithi188 ceph-mon[129553]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:50:40.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:40 smithi158 ceph-mon[128925]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2023-07-20T02:50:40.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:40 smithi158 ceph-mon[128925]: osdmap e77: 12 total, 11 up, 12 in 2023-07-20T02:50:40.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:40 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T02:50:40.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:40 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T02:50:40.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:40 smithi158 ceph-mon[128925]: Detected new or changed devices on smithi188 2023-07-20T02:50:40.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:40 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:40.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:40 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:40.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:40 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:50:40.642 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:40 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:50:40.642 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:40 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:50:40.642 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:40 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-20T02:50:40.642 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:40 smithi158 ceph-mon[128925]: Adjusting osd_memory_target on smithi188 to 4292M 2023-07-20T02:50:40.642 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:40 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:40.642 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:40 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:50:40.642 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:40 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/4077569660' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T02:50:40.643 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:40 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:50:40.643 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:40 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:40.643 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:40 smithi158 ceph-mon[128925]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-20T02:50:40.716 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph osd stat -f json 2023-07-20T02:50:41.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:41 smithi007 ceph-mon[126353]: purged_snaps scrub starts 2023-07-20T02:50:41.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:41 smithi007 ceph-mon[126353]: purged_snaps scrub ok 2023-07-20T02:50:41.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:41 smithi007 ceph-mon[126353]: osd.11 [v2:172.21.15.188:6824/1230196525,v1:172.21.15.188:6825/1230196525] boot 2023-07-20T02:50:41.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:41 smithi007 ceph-mon[126353]: osdmap e78: 12 total, 12 up, 12 in 2023-07-20T02:50:41.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:41 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T02:50:41.460 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:50:41.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:41 smithi188 ceph-mon[129553]: purged_snaps scrub starts 2023-07-20T02:50:41.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:41 smithi188 ceph-mon[129553]: purged_snaps scrub ok 2023-07-20T02:50:41.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:41 smithi188 ceph-mon[129553]: osd.11 [v2:172.21.15.188:6824/1230196525,v1:172.21.15.188:6825/1230196525] boot 2023-07-20T02:50:41.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:41 smithi188 ceph-mon[129553]: osdmap e78: 12 total, 12 up, 12 in 2023-07-20T02:50:41.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:41 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T02:50:41.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:41 smithi158 ceph-mon[128925]: purged_snaps scrub starts 2023-07-20T02:50:41.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:41 smithi158 ceph-mon[128925]: purged_snaps scrub ok 2023-07-20T02:50:41.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:41 smithi158 ceph-mon[128925]: osd.11 [v2:172.21.15.188:6824/1230196525,v1:172.21.15.188:6825/1230196525] boot 2023-07-20T02:50:41.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:41 smithi158 ceph-mon[128925]: osdmap e78: 12 total, 12 up, 12 in 2023-07-20T02:50:41.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:41 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T02:50:42.271 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:42 smithi007 ceph-mon[126353]: osdmap e79: 12 total, 12 up, 12 in 2023-07-20T02:50:42.271 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:42 smithi007 ceph-mon[126353]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:50:42.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:42 smithi188 ceph-mon[129553]: osdmap e79: 12 total, 12 up, 12 in 2023-07-20T02:50:42.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:42 smithi188 ceph-mon[129553]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:50:42.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:42 smithi158 ceph-mon[128925]: osdmap e79: 12 total, 12 up, 12 in 2023-07-20T02:50:42.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:42 smithi158 ceph-mon[128925]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:50:42.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.927+0000 7f70eea6b700 1 -- 172.21.15.7:0/3082274448 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70e8101a00 msgr2=0x7f70e8101e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:42.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.927+0000 7f70eea6b700 1 --2- 172.21.15.7:0/3082274448 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70e8101a00 0x7f70e8101e60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f70d40099c0 tx=0x7f70d4009cd0 comp rx=0 tx=0).stop 2023-07-20T02:50:42.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.927+0000 7f70eea6b700 1 -- 172.21.15.7:0/3082274448 shutdown_connections 2023-07-20T02:50:42.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.927+0000 7f70eea6b700 1 --2- 172.21.15.7:0/3082274448 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70e81023a0 0x7f70e8108c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:42.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.927+0000 7f70eea6b700 1 --2- 172.21.15.7:0/3082274448 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70e8100800 0x7f70e8100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:42.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.927+0000 7f70eea6b700 1 --2- 172.21.15.7:0/3082274448 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70e8101a00 0x7f70e8101e60 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:42.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.927+0000 7f70eea6b700 1 -- 172.21.15.7:0/3082274448 >> 172.21.15.7:0/3082274448 conn(0x7f70e80fbd60 msgr2=0x7f70e80fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:50:42.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.927+0000 7f70eea6b700 1 -- 172.21.15.7:0/3082274448 shutdown_connections 2023-07-20T02:50:42.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.927+0000 7f70eea6b700 1 -- 172.21.15.7:0/3082274448 wait complete. 2023-07-20T02:50:42.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.928+0000 7f70eea6b700 1 Processor -- start 2023-07-20T02:50:42.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.928+0000 7f70eea6b700 1 -- start start 2023-07-20T02:50:42.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.929+0000 7f70eea6b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70e8100800 0x7f70e8196510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:42.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.929+0000 7f70eea6b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70e8101a00 0x7f70e8196a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:42.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.929+0000 7f70eea6b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70e81023a0 0x7f70e819dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:42.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.929+0000 7f70eea6b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e819e0a0 con 0x7f70e8101a00 2023-07-20T02:50:42.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.929+0000 7f70eea6b700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e819e1e0 con 0x7f70e81023a0 2023-07-20T02:50:42.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.929+0000 7f70eea6b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e819e370 con 0x7f70e8100800 2023-07-20T02:50:42.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.929+0000 7f70e7fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70e8101a00 0x7f70e8196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:42.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.929+0000 7f70e7fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70e8101a00 0x7f70e8196a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55798/0 (socket says 172.21.15.7:55798) 2023-07-20T02:50:42.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.929+0000 7f70e7fff700 1 -- 172.21.15.7:0/86926612 learned_addr learned my addr 172.21.15.7:0/86926612 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:50:42.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.929+0000 7f70ec807700 1 --2- 172.21.15.7:0/86926612 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70e8100800 0x7f70e8196510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:42.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.929+0000 7f70ed008700 1 --2- 172.21.15.7:0/86926612 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70e81023a0 0x7f70e819dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:42.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.929+0000 7f70e7fff700 1 -- 172.21.15.7:0/86926612 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70e8100800 msgr2=0x7f70e8196510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:42.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.929+0000 7f70e7fff700 1 --2- 172.21.15.7:0/86926612 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70e8100800 0x7f70e8196510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:42.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.929+0000 7f70e7fff700 1 -- 172.21.15.7:0/86926612 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70e81023a0 msgr2=0x7f70e819dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:42.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.929+0000 7f70e7fff700 1 --2- 172.21.15.7:0/86926612 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70e81023a0 0x7f70e819dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:42.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.929+0000 7f70e7fff700 1 -- 172.21.15.7:0/86926612 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d40095f0 con 0x7f70e8101a00 2023-07-20T02:50:42.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.929+0000 7f70ed008700 1 --2- 172.21.15.7:0/86926612 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70e81023a0 0x7f70e819dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T02:50:42.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.930+0000 7f70e7fff700 1 --2- 172.21.15.7:0/86926612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70e8101a00 0x7f70e8196a50 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f70d40095c0 tx=0x7f70d400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:50:42.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.930+0000 7f70e5ffb700 1 -- 172.21.15.7:0/86926612 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d401d070 con 0x7f70e8101a00 2023-07-20T02:50:42.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.930+0000 7f70e5ffb700 1 -- 172.21.15.7:0/86926612 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f70d4005050 con 0x7f70e8101a00 2023-07-20T02:50:42.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.930+0000 7f70eea6b700 1 -- 172.21.15.7:0/86926612 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70e819e5f0 con 0x7f70e8101a00 2023-07-20T02:50:42.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.930+0000 7f70e5ffb700 1 -- 172.21.15.7:0/86926612 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d4004be0 con 0x7f70e8101a00 2023-07-20T02:50:42.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.930+0000 7f70eea6b700 1 -- 172.21.15.7:0/86926612 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70e819ea00 con 0x7f70e8101a00 2023-07-20T02:50:42.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.931+0000 7f70eea6b700 1 -- 172.21.15.7:0/86926612 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70e810c050 con 0x7f70e8101a00 2023-07-20T02:50:42.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.933+0000 7f70e5ffb700 1 -- 172.21.15.7:0/86926612 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f70d40051c0 con 0x7f70e8101a00 2023-07-20T02:50:42.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.934+0000 7f70e5ffb700 1 --2- 172.21.15.7:0/86926612 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f70d80a1150 0x7f70d80a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:42.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.935+0000 7f70e5ffb700 1 -- 172.21.15.7:0/86926612 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f70d40cd420 con 0x7f70e8101a00 2023-07-20T02:50:42.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.935+0000 7f70ec807700 1 --2- 172.21.15.7:0/86926612 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f70d80a1150 0x7f70d80a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:42.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.935+0000 7f70ec807700 1 --2- 172.21.15.7:0/86926612 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f70d80a1150 0x7f70d80a3610 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f70e8101860 tx=0x7f70dc007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:50:42.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:42.938+0000 7f70e5ffb700 1 -- 172.21.15.7:0/86926612 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f70d4097000 con 0x7f70e8101a00 2023-07-20T02:50:43.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:43.144+0000 7f70eea6b700 1 -- 172.21.15.7:0/86926612 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f70e81077a0 con 0x7f70e8101a00 2023-07-20T02:50:43.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:43.144+0000 7f70e5ffb700 1 -- 172.21.15.7:0/86926612 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v79) v1 ==== 74+0+133 (secure 0 0 0) 0x7f70d409a620 con 0x7f70e8101a00 2023-07-20T02:50:43.157 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:50:43.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:43.147+0000 7f70eea6b700 1 -- 172.21.15.7:0/86926612 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f70d80a1150 msgr2=0x7f70d80a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:43.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:43.147+0000 7f70eea6b700 1 --2- 172.21.15.7:0/86926612 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f70d80a1150 0x7f70d80a3610 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f70e8101860 tx=0x7f70dc007400 comp rx=0 tx=0).stop 2023-07-20T02:50:43.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:43.147+0000 7f70eea6b700 1 -- 172.21.15.7:0/86926612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70e8101a00 msgr2=0x7f70e8196a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:43.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:43.147+0000 7f70eea6b700 1 --2- 172.21.15.7:0/86926612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70e8101a00 0x7f70e8196a50 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f70d40095c0 tx=0x7f70d400f690 comp rx=0 tx=0).stop 2023-07-20T02:50:43.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:43.147+0000 7f70eea6b700 1 -- 172.21.15.7:0/86926612 shutdown_connections 2023-07-20T02:50:43.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:43.147+0000 7f70eea6b700 1 --2- 172.21.15.7:0/86926612 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f70e81023a0 0x7f70e819dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:43.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:43.147+0000 7f70eea6b700 1 --2- 172.21.15.7:0/86926612 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70e8100800 0x7f70e8196510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:43.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:43.147+0000 7f70eea6b700 1 --2- 172.21.15.7:0/86926612 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f70d80a1150 0x7f70d80a3610 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:43.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:43.147+0000 7f70eea6b700 1 --2- 172.21.15.7:0/86926612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70e8101a00 0x7f70e8196a50 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:43.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:43.147+0000 7f70eea6b700 1 -- 172.21.15.7:0/86926612 >> 172.21.15.7:0/86926612 conn(0x7f70e80fbd60 msgr2=0x7f70e80fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:50:43.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:43.147+0000 7f70eea6b700 1 -- 172.21.15.7:0/86926612 shutdown_connections 2023-07-20T02:50:43.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:43.147+0000 7f70eea6b700 1 -- 172.21.15.7:0/86926612 wait complete. 2023-07-20T02:50:43.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:43 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/86926612' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T02:50:43.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:43 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/86926612' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T02:50:43.640 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":79,"num_osds":12,"num_up_osds":12,"osd_up_since":1689821440,"num_in_osds":12,"osd_in_since":1689821421,"num_remapped_pgs":0} 2023-07-20T02:50:43.641 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph osd dump --format=json 2023-07-20T02:50:43.642 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:43 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/86926612' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T02:50:44.373 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:50:44.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:44 smithi158 ceph-mon[128925]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:50:44.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:44 smithi007 ceph-mon[126353]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:50:44.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:44 smithi188 ceph-mon[129553]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:50:45.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.857+0000 7fb8c602e700 1 -- 172.21.15.7:0/4004432398 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8c0100070 msgr2=0x7fb8c010c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:45.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.857+0000 7fb8c602e700 1 --2- 172.21.15.7:0/4004432398 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8c0100070 0x7fb8c010c510 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb8b800b0a0 tx=0x7fb8b800b3b0 comp rx=0 tx=0).stop 2023-07-20T02:50:45.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.857+0000 7fb8c602e700 1 -- 172.21.15.7:0/4004432398 shutdown_connections 2023-07-20T02:50:45.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.857+0000 7fb8c602e700 1 --2- 172.21.15.7:0/4004432398 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8c0100070 0x7fb8c010c510 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:45.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.857+0000 7fb8c602e700 1 --2- 172.21.15.7:0/4004432398 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb8c0104580 0x7fb8c0104960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:45.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.857+0000 7fb8c602e700 1 --2- 172.21.15.7:0/4004432398 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8c00ff6b0 0x7fb8c00ffb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:45.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.857+0000 7fb8c602e700 1 -- 172.21.15.7:0/4004432398 >> 172.21.15.7:0/4004432398 conn(0x7fb8c0076770 msgr2=0x7fb8c0076b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:50:45.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.857+0000 7fb8c602e700 1 -- 172.21.15.7:0/4004432398 shutdown_connections 2023-07-20T02:50:45.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.857+0000 7fb8c602e700 1 -- 172.21.15.7:0/4004432398 wait complete. 2023-07-20T02:50:45.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.858+0000 7fb8c602e700 1 Processor -- start 2023-07-20T02:50:45.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.859+0000 7fb8c602e700 1 -- start start 2023-07-20T02:50:45.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.859+0000 7fb8c602e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8c00ff6b0 0x7fb8c010ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:45.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.859+0000 7fb8c602e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb8c0100070 0x7fb8c010f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:45.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.859+0000 7fb8c602e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8c0104580 0x7fb8c010f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:45.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.859+0000 7fb8c602e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c01a4de0 con 0x7fb8c00ff6b0 2023-07-20T02:50:45.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.859+0000 7fb8c602e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c01a4f50 con 0x7fb8c0104580 2023-07-20T02:50:45.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.859+0000 7fb8c602e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c01a5110 con 0x7fb8c0100070 2023-07-20T02:50:45.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.859+0000 7fb8bf7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8c00ff6b0 0x7fb8c010ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:45.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.859+0000 7fb8bf7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8c00ff6b0 0x7fb8c010ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55824/0 (socket says 172.21.15.7:55824) 2023-07-20T02:50:45.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.859+0000 7fb8bf7fe700 1 -- 172.21.15.7:0/2533237239 learned_addr learned my addr 172.21.15.7:0/2533237239 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:50:45.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.859+0000 7fb8beffd700 1 --2- 172.21.15.7:0/2533237239 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb8c0100070 0x7fb8c010f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:45.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.860+0000 7fb8bffff700 1 --2- 172.21.15.7:0/2533237239 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8c0104580 0x7fb8c010f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:45.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.860+0000 7fb8bf7fe700 1 -- 172.21.15.7:0/2533237239 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb8c0100070 msgr2=0x7fb8c010f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:45.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.860+0000 7fb8bf7fe700 1 --2- 172.21.15.7:0/2533237239 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb8c0100070 0x7fb8c010f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:45.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.860+0000 7fb8bf7fe700 1 -- 172.21.15.7:0/2533237239 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8c0104580 msgr2=0x7fb8c010f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:45.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.860+0000 7fb8bf7fe700 1 --2- 172.21.15.7:0/2533237239 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8c0104580 0x7fb8c010f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:45.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.860+0000 7fb8bf7fe700 1 -- 172.21.15.7:0/2533237239 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b8009d00 con 0x7fb8c00ff6b0 2023-07-20T02:50:45.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.860+0000 7fb8bffff700 1 --2- 172.21.15.7:0/2533237239 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8c0104580 0x7fb8c010f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T02:50:45.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.860+0000 7fb8bf7fe700 1 --2- 172.21.15.7:0/2533237239 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8c00ff6b0 0x7fb8c010ee70 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb8b000d950 tx=0x7fb8b000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:50:45.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.860+0000 7fb8bcff9700 1 -- 172.21.15.7:0/2533237239 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b0009940 con 0x7fb8c00ff6b0 2023-07-20T02:50:45.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.860+0000 7fb8bcff9700 1 -- 172.21.15.7:0/2533237239 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb8b0010460 con 0x7fb8c00ff6b0 2023-07-20T02:50:45.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.860+0000 7fb8c602e700 1 -- 172.21.15.7:0/2533237239 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8c01a53f0 con 0x7fb8c00ff6b0 2023-07-20T02:50:45.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.860+0000 7fb8bcff9700 1 -- 172.21.15.7:0/2533237239 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b000f620 con 0x7fb8c00ff6b0 2023-07-20T02:50:45.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.860+0000 7fb8c602e700 1 -- 172.21.15.7:0/2533237239 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8c01a58f0 con 0x7fb8c00ff6b0 2023-07-20T02:50:45.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.862+0000 7fb8c602e700 1 -- 172.21.15.7:0/2533237239 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8c004e470 con 0x7fb8c00ff6b0 2023-07-20T02:50:45.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.864+0000 7fb8bcff9700 1 -- 172.21.15.7:0/2533237239 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fb8b00108d0 con 0x7fb8c00ff6b0 2023-07-20T02:50:45.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.866+0000 7fb8bcff9700 1 --2- 172.21.15.7:0/2533237239 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fb8a80a1150 0x7fb8a80a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:45.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.866+0000 7fb8bcff9700 1 -- 172.21.15.7:0/2533237239 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb8b00cc330 con 0x7fb8c00ff6b0 2023-07-20T02:50:45.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.866+0000 7fb8beffd700 1 --2- 172.21.15.7:0/2533237239 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fb8a80a1150 0x7fb8a80a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:45.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.867+0000 7fb8beffd700 1 --2- 172.21.15.7:0/2533237239 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fb8a80a1150 0x7fb8a80a3610 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb8c0110660 tx=0x7fb8b400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:50:45.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:45.870+0000 7fb8bcff9700 1 -- 172.21.15.7:0/2533237239 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb8b0095e80 con 0x7fb8c00ff6b0 2023-07-20T02:50:46.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:46.079+0000 7fb8c602e700 1 -- 172.21.15.7:0/2533237239 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8c01103a0 con 0x7fb8c00ff6b0 2023-07-20T02:50:46.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:46.080+0000 7fb8bcff9700 1 -- 172.21.15.7:0/2533237239 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+18389 (secure 0 0 0) 0x7fb8b00b9070 con 0x7fb8c00ff6b0 2023-07-20T02:50:46.093 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:50:46.094 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":79,"fsid":"1b04643c-26a7-11ee-9b34-001a4aab830c","created":"2023-07-20T02:43:56.544719+0000","modified":"2023-07-20T02:50:41.133457+0000","last_up_change":"2023-07-20T02:50:40.126926+0000","last_in_change":"2023-07-20T02:50:21.875020+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T02:47:12.055539+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad6c6339-6e23-4c21-9fde-2dbecdafdb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6803","nonce":3322992091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6805","nonce":3322992091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6809","nonce":3322992091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6807","nonce":3322992091}]},"public_addr":"172.21.15.7:6803/3322992091","cluster_addr":"172.21.15.7:6805/3322992091","heartbeat_back_addr":"172.21.15.7:6809/3322992091","heartbeat_front_addr":"172.21.15.7:6807/3322992091","state":["exists","up"]},{"osd":1,"uuid":"cad207b3-a1ea-4f07-bd94-9b11fd573a69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6801","nonce":2796771612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6803","nonce":2796771612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6807","nonce":2796771612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6805","nonce":2796771612}]},"public_addr":"172.21.15.158:6801/2796771612","cluster_addr":"172.21.15.158:6803/2796771612","heartbeat_back_addr":"172.21.15.158:6807/2796771612","heartbeat_front_addr":"172.21.15.158:6805/2796771612","state":["exists","up"]},{"osd":2,"uuid":"2c777fe6-d003-49d0-af37-f63fca37ff06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6801","nonce":2928307258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6803","nonce":2928307258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6807","nonce":2928307258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6805","nonce":2928307258}]},"public_addr":"172.21.15.188:6801/2928307258","cluster_addr":"172.21.15.188:6803/2928307258","heartbeat_back_addr":"172.21.15.188:6807/2928307258","heartbeat_front_addr":"172.21.15.188:6805/2928307258","state":["exists","up"]},{"osd":3,"uuid":"59e29965-018f-4eb6-adbd-1a9e1df84245","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6811","nonce":2192953827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6813","nonce":2192953827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6817","nonce":2192953827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6815","nonce":2192953827}]},"public_addr":"172.21.15.7:6811/2192953827","cluster_addr":"172.21.15.7:6813/2192953827","heartbeat_back_addr":"172.21.15.7:6817/2192953827","heartbeat_front_addr":"172.21.15.7:6815/2192953827","state":["exists","up"]},{"osd":4,"uuid":"e2453384-3b09-4f2a-bea7-172d8fc466d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6809","nonce":3330591132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6810","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6811","nonce":3330591132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6814","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6815","nonce":3330591132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6813","nonce":3330591132}]},"public_addr":"172.21.15.158:6809/3330591132","cluster_addr":"172.21.15.158:6811/3330591132","heartbeat_back_addr":"172.21.15.158:6815/3330591132","heartbeat_front_addr":"172.21.15.158:6813/3330591132","state":["exists","up"]},{"osd":5,"uuid":"c3c04c13-0ce5-440d-9e62-0e301ad9f771","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6809","nonce":3142012805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6811","nonce":3142012805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6815","nonce":3142012805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6813","nonce":3142012805}]},"public_addr":"172.21.15.188:6809/3142012805","cluster_addr":"172.21.15.188:6811/3142012805","heartbeat_back_addr":"172.21.15.188:6815/3142012805","heartbeat_front_addr":"172.21.15.188:6813/3142012805","state":["exists","up"]},{"osd":6,"uuid":"82ef2d33-1045-41f1-bf69-b03404bcace8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6819","nonce":859560576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6821","nonce":859560576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6825","nonce":859560576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6823","nonce":859560576}]},"public_addr":"172.21.15.7:6819/859560576","cluster_addr":"172.21.15.7:6821/859560576","heartbeat_back_addr":"172.21.15.7:6825/859560576","heartbeat_front_addr":"172.21.15.7:6823/859560576","state":["exists","up"]},{"osd":7,"uuid":"020ec859-3fc2-4da2-a0dc-10530a4a366c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6817","nonce":2578758635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6819","nonce":2578758635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6822","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6823","nonce":2578758635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6821","nonce":2578758635}]},"public_addr":"172.21.15.158:6817/2578758635","cluster_addr":"172.21.15.158:6819/2578758635","heartbeat_back_addr":"172.21.15.158:6823/2578758635","heartbeat_front_addr":"172.21.15.158:6821/2578758635","state":["exists","up"]},{"osd":8,"uuid":"4081496a-45ee-4b25-8f17-a23665521187","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6817","nonce":3020824843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6819","nonce":3020824843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6823","nonce":3020824843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6821","nonce":3020824843}]},"public_addr":"172.21.15.188:6817/3020824843","cluster_addr":"172.21.15.188:6819/3020824843","heartbeat_back_addr":"172.21.15.188:6823/3020824843","heartbeat_front_addr":"172.21.15.188:6821/3020824843","state":["exists","up"]},{"osd":9,"uuid":"723580ae-efe3-4d86-95fd-d7a4c0631425","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6827","nonce":1612464825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6829","nonce":1612464825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6833","nonce":1612464825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6831","nonce":1612464825}]},"public_addr":"172.21.15.7:6827/1612464825","cluster_addr":"172.21.15.7:6829/1612464825","heartbeat_back_addr":"172.21.15.7:6833/1612464825","heartbeat_front_addr":"172.21.15.7:6831/1612464825","state":["exists","up"]},{"osd":10,"uuid":"c2651548-fe46-4d2d-b911-278036412ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6825","nonce":1653166210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6827","nonce":1653166210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6831","nonce":1653166210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6829","nonce":1653166210}]},"public_addr":"172.21.15.158:6825/1653166210","cluster_addr":"172.21.15.158:6827/1653166210","heartbeat_back_addr":"172.21.15.158:6831/1653166210","heartbeat_front_addr":"172.21.15.158:6829/1653166210","state":["exists","up"]},{"osd":11,"uuid":"f7262bcb-4243-403a-a321-fc2bdf1d4d25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6825","nonce":1230196525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6827","nonce":1230196525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6831","nonce":1230196525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6829","nonce":1230196525}]},"public_addr":"172.21.15.188:6825/1230196525","cluster_addr":"172.21.15.188:6827/1230196525","heartbeat_back_addr":"172.21.15.188:6831/1230196525","heartbeat_front_addr":"172.21.15.188:6829/1230196525","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:46:26.803832+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:46:48.726253+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:47:09.744123+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:47:36.907588+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:47:57.696685+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:48:19.120806+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:48:45.313043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:49:06.686384+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:49:28.475078+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:49:54.984033+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:50:16.541259+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:50:38.221746+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/3892248290":"2023-07-21T02:44:53.458649+0000","172.21.15.7:0/2373058722":"2023-07-21T02:44:53.458649+0000","172.21.15.7:6801/2926310425":"2023-07-21T02:44:53.458649+0000","172.21.15.7:0/4222569264":"2023-07-21T02:44:25.804258+0000","172.21.15.7:0/1357065986":"2023-07-21T02:44:25.804258+0000","172.21.15.7:0/884368977":"2023-07-21T02:44:53.458649+0000","172.21.15.7:6801/1224472946":"2023-07-21T02:44:25.804258+0000","172.21.15.7:0/1459293748":"2023-07-21T02:44:25.804258+0000","172.21.15.7:6800/1224472946":"2023-07-21T02:44:25.804258+0000","172.21.15.7:6800/2926310425":"2023-07-21T02:44:53.458649+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T02:50:46.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:46.083+0000 7fb8c602e700 1 -- 172.21.15.7:0/2533237239 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fb8a80a1150 msgr2=0x7fb8a80a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:46.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:46.083+0000 7fb8c602e700 1 --2- 172.21.15.7:0/2533237239 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fb8a80a1150 0x7fb8a80a3610 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb8c0110660 tx=0x7fb8b400b410 comp rx=0 tx=0).stop 2023-07-20T02:50:46.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:46.083+0000 7fb8c602e700 1 -- 172.21.15.7:0/2533237239 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8c00ff6b0 msgr2=0x7fb8c010ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:46.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:46.083+0000 7fb8c602e700 1 --2- 172.21.15.7:0/2533237239 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8c00ff6b0 0x7fb8c010ee70 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb8b000d950 tx=0x7fb8b000dc60 comp rx=0 tx=0).stop 2023-07-20T02:50:46.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:46.083+0000 7fb8c602e700 1 -- 172.21.15.7:0/2533237239 shutdown_connections 2023-07-20T02:50:46.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:46.083+0000 7fb8c602e700 1 --2- 172.21.15.7:0/2533237239 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb8c0104580 0x7fb8c010f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:46.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:46.083+0000 7fb8c602e700 1 --2- 172.21.15.7:0/2533237239 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fb8a80a1150 0x7fb8a80a3610 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:46.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:46.083+0000 7fb8c602e700 1 --2- 172.21.15.7:0/2533237239 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8c00ff6b0 0x7fb8c010ee70 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:46.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:46.083+0000 7fb8c602e700 1 --2- 172.21.15.7:0/2533237239 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb8c0100070 0x7fb8c010f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:46.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:46.083+0000 7fb8c602e700 1 -- 172.21.15.7:0/2533237239 >> 172.21.15.7:0/2533237239 conn(0x7fb8c0076770 msgr2=0x7fb8c00fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:50:46.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:46.083+0000 7fb8c602e700 1 -- 172.21.15.7:0/2533237239 shutdown_connections 2023-07-20T02:50:46.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:46.083+0000 7fb8c602e700 1 -- 172.21.15.7:0/2533237239 wait complete. 2023-07-20T02:50:46.552 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-20T02:47:12.055539+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-20T02:50:46.553 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph osd pool get .mgr pg_num 2023-07-20T02:50:46.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:46 smithi158 ceph-mon[128925]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:50:46.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:46 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/2533237239' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:50:46.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:46 smithi007 ceph-mon[126353]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:50:46.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:46 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/2533237239' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:50:46.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:46 smithi188 ceph-mon[129553]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:50:46.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:46 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/2533237239' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:50:47.287 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:50:48.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:48 smithi158 ceph-mon[128925]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:50:48.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:48 smithi007 ceph-mon[126353]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:50:48.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:48 smithi188 ceph-mon[129553]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:50:48.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.779+0000 7fc5feb00700 1 -- 172.21.15.7:0/1746942506 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5f8106790 msgr2=0x7fc5f8106b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:48.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.779+0000 7fc5feb00700 1 --2- 172.21.15.7:0/1746942506 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5f8106790 0x7fc5f8106b70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec009990 tx=0x7fc5ec009ca0 comp rx=0 tx=0).stop 2023-07-20T02:50:48.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.780+0000 7fc5feb00700 1 -- 172.21.15.7:0/1746942506 shutdown_connections 2023-07-20T02:50:48.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.780+0000 7fc5feb00700 1 --2- 172.21.15.7:0/1746942506 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5f8101150 0x7fc5f810d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:48.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.780+0000 7fc5feb00700 1 --2- 172.21.15.7:0/1746942506 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5f8106790 0x7fc5f8106b70 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:48.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.780+0000 7fc5feb00700 1 --2- 172.21.15.7:0/1746942506 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5f8100790 0x7fc5f8100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:48.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.780+0000 7fc5feb00700 1 -- 172.21.15.7:0/1746942506 >> 172.21.15.7:0/1746942506 conn(0x7fc5f80fc260 msgr2=0x7fc5f80fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:50:48.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.780+0000 7fc5feb00700 1 -- 172.21.15.7:0/1746942506 shutdown_connections 2023-07-20T02:50:48.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.780+0000 7fc5feb00700 1 -- 172.21.15.7:0/1746942506 wait complete. 2023-07-20T02:50:48.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.781+0000 7fc5feb00700 1 Processor -- start 2023-07-20T02:50:48.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.781+0000 7fc5feb00700 1 -- start start 2023-07-20T02:50:48.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.781+0000 7fc5feb00700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5f8100790 0x7fc5f8072a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:48.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.781+0000 7fc5feb00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5f8101150 0x7fc5f8072fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:48.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.781+0000 7fc5feb00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5f8106790 0x7fc5f8073530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:48.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.781+0000 7fc5feb00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f81a4d40 con 0x7fc5f8106790 2023-07-20T02:50:48.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.781+0000 7fc5feb00700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f81a4eb0 con 0x7fc5f8100790 2023-07-20T02:50:48.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.781+0000 7fc5feb00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f81a5070 con 0x7fc5f8101150 2023-07-20T02:50:48.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.781+0000 7fc5fd09d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5f8106790 0x7fc5f8073530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:48.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.782+0000 7fc5fd09d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5f8106790 0x7fc5f8073530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55836/0 (socket says 172.21.15.7:55836) 2023-07-20T02:50:48.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.782+0000 7fc5fd09d700 1 -- 172.21.15.7:0/1746100734 learned_addr learned my addr 172.21.15.7:0/1746100734 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:50:48.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.782+0000 7fc5fc89c700 1 --2- 172.21.15.7:0/1746100734 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5f8100790 0x7fc5f8072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:48.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.782+0000 7fc5f7fff700 1 --2- 172.21.15.7:0/1746100734 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5f8101150 0x7fc5f8072fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:48.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.782+0000 7fc5fd09d700 1 -- 172.21.15.7:0/1746100734 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5f8101150 msgr2=0x7fc5f8072fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:48.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.782+0000 7fc5fd09d700 1 --2- 172.21.15.7:0/1746100734 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5f8101150 0x7fc5f8072fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:48.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.782+0000 7fc5fd09d700 1 -- 172.21.15.7:0/1746100734 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5f8100790 msgr2=0x7fc5f8072a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:48.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.782+0000 7fc5fd09d700 1 --2- 172.21.15.7:0/1746100734 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5f8100790 0x7fc5f8072a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:48.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.782+0000 7fc5fd09d700 1 -- 172.21.15.7:0/1746100734 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5ec0095f0 con 0x7fc5f8106790 2023-07-20T02:50:48.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.782+0000 7fc5fd09d700 1 --2- 172.21.15.7:0/1746100734 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5f8106790 0x7fc5f8073530 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc5f0007ee0 tx=0x7fc5f000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:50:48.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.782+0000 7fc5f5ffb700 1 -- 172.21.15.7:0/1746100734 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f000dd20 con 0x7fc5f8106790 2023-07-20T02:50:48.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.782+0000 7fc5feb00700 1 -- 172.21.15.7:0/1746100734 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5f81a5350 con 0x7fc5f8106790 2023-07-20T02:50:48.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.782+0000 7fc5f5ffb700 1 -- 172.21.15.7:0/1746100734 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc5f000f040 con 0x7fc5f8106790 2023-07-20T02:50:48.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.783+0000 7fc5f5ffb700 1 -- 172.21.15.7:0/1746100734 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f00048a0 con 0x7fc5f8106790 2023-07-20T02:50:48.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.783+0000 7fc5feb00700 1 -- 172.21.15.7:0/1746100734 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5f81a5800 con 0x7fc5f8106790 2023-07-20T02:50:48.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.784+0000 7fc5feb00700 1 -- 172.21.15.7:0/1746100734 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5f804e470 con 0x7fc5f8106790 2023-07-20T02:50:48.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.786+0000 7fc5f5ffb700 1 -- 172.21.15.7:0/1746100734 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fc5f0003a00 con 0x7fc5f8106790 2023-07-20T02:50:48.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.787+0000 7fc5f5ffb700 1 --2- 172.21.15.7:0/1746100734 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fc5e80a1150 0x7fc5e80a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:48.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.787+0000 7fc5f5ffb700 1 -- 172.21.15.7:0/1746100734 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc5f00cbd20 con 0x7fc5f8106790 2023-07-20T02:50:48.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.787+0000 7fc5fc89c700 1 --2- 172.21.15.7:0/1746100734 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fc5e80a1150 0x7fc5e80a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:48.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.788+0000 7fc5fc89c700 1 --2- 172.21.15.7:0/1746100734 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fc5e80a1150 0x7fc5e80a3610 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec00b2e0 tx=0x7fc5ec005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:50:48.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.791+0000 7fc5f5ffb700 1 -- 172.21.15.7:0/1746100734 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc5f004e780 con 0x7fc5f8106790 2023-07-20T02:50:49.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.994+0000 7fc5feb00700 1 -- 172.21.15.7:0/1746100734 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc5f81a5b50 con 0x7fc5f8106790 2023-07-20T02:50:49.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.995+0000 7fc5f5ffb700 1 -- 172.21.15.7:0/1746100734 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc5f0095870 con 0x7fc5f8106790 2023-07-20T02:50:49.008 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2023-07-20T02:50:49.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.997+0000 7fc5feb00700 1 -- 172.21.15.7:0/1746100734 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fc5e80a1150 msgr2=0x7fc5e80a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:49.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.997+0000 7fc5feb00700 1 --2- 172.21.15.7:0/1746100734 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fc5e80a1150 0x7fc5e80a3610 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec00b2e0 tx=0x7fc5ec005fd0 comp rx=0 tx=0).stop 2023-07-20T02:50:49.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.997+0000 7fc5feb00700 1 -- 172.21.15.7:0/1746100734 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5f8106790 msgr2=0x7fc5f8073530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:49.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.997+0000 7fc5feb00700 1 --2- 172.21.15.7:0/1746100734 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5f8106790 0x7fc5f8073530 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc5f0007ee0 tx=0x7fc5f000d3b0 comp rx=0 tx=0).stop 2023-07-20T02:50:49.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.997+0000 7fc5feb00700 1 -- 172.21.15.7:0/1746100734 shutdown_connections 2023-07-20T02:50:49.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.997+0000 7fc5feb00700 1 --2- 172.21.15.7:0/1746100734 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fc5e80a1150 0x7fc5e80a3610 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:49.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.997+0000 7fc5feb00700 1 --2- 172.21.15.7:0/1746100734 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5f8106790 0x7fc5f8073530 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:49.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.997+0000 7fc5feb00700 1 --2- 172.21.15.7:0/1746100734 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5f8100790 0x7fc5f8072a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:49.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.997+0000 7fc5feb00700 1 --2- 172.21.15.7:0/1746100734 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5f8101150 0x7fc5f8072fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:49.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.997+0000 7fc5feb00700 1 -- 172.21.15.7:0/1746100734 >> 172.21.15.7:0/1746100734 conn(0x7fc5f80fc260 msgr2=0x7fc5f80fc890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:50:49.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.998+0000 7fc5feb00700 1 -- 172.21.15.7:0/1746100734 shutdown_connections 2023-07-20T02:50:49.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:48.998+0000 7fc5feb00700 1 -- 172.21.15.7:0/1746100734 wait complete. 2023-07-20T02:50:49.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:49 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/1746100734' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T02:50:49.430 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:49 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/1746100734' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T02:50:49.462 INFO:tasks.cephadm:Adding mds.a on smithi007 2023-07-20T02:50:49.462 INFO:tasks.cephadm:Adding mds.d on smithi007 2023-07-20T02:50:49.463 INFO:tasks.cephadm:Adding mds.g on smithi007 2023-07-20T02:50:49.463 INFO:tasks.cephadm:Adding mds.j on smithi007 2023-07-20T02:50:49.463 INFO:tasks.cephadm:Adding mds.b on smithi158 2023-07-20T02:50:49.463 INFO:tasks.cephadm:Adding mds.e on smithi158 2023-07-20T02:50:49.463 INFO:tasks.cephadm:Adding mds.h on smithi158 2023-07-20T02:50:49.464 INFO:tasks.cephadm:Adding mds.k on smithi158 2023-07-20T02:50:49.464 INFO:tasks.cephadm:Adding mds.c on smithi188 2023-07-20T02:50:49.464 INFO:tasks.cephadm:Adding mds.f on smithi188 2023-07-20T02:50:49.464 INFO:tasks.cephadm:Adding mds.i on smithi188 2023-07-20T02:50:49.464 INFO:tasks.cephadm:Adding mds.l on smithi188 2023-07-20T02:50:49.465 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph orch apply mds all '12;smithi007=a;smithi007=d;smithi007=g;smithi007=j;smithi158=b;smithi158=e;smithi158=h;smithi158=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l' 2023-07-20T02:50:49.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:49 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/1746100734' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T02:50:50.032 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T02:50:50.435 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:50 smithi188 ceph-mon[129553]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:50:50.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:50 smithi007 ceph-mon[126353]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:50:50.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:50 smithi158 ceph-mon[128925]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:50:51.362 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.348+0000 7f92274ae700 1 -- 172.21.15.188:0/767916093 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9220100dd0 msgr2=0x7f92201011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:51.362 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.348+0000 7f92274ae700 1 --2- 172.21.15.188:0/767916093 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9220100dd0 0x7f92201011f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9210009990 tx=0x7f9210009ca0 comp rx=0 tx=0).stop 2023-07-20T02:50:51.362 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.349+0000 7f92274ae700 1 -- 172.21.15.188:0/767916093 shutdown_connections 2023-07-20T02:50:51.362 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.349+0000 7f92274ae700 1 --2- 172.21.15.188:0/767916093 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9220100360 0x7f9220107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:51.363 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.349+0000 7f92274ae700 1 --2- 172.21.15.188:0/767916093 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9220100dd0 0x7f92201011f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:51.363 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.349+0000 7f92274ae700 1 --2- 172.21.15.188:0/767916093 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92200ff9a0 0x7f92200ffe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:51.363 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.349+0000 7f92274ae700 1 -- 172.21.15.188:0/767916093 >> 172.21.15.188:0/767916093 conn(0x7f92200fa450 msgr2=0x7f92200fc8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:50:51.363 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.349+0000 7f92274ae700 1 -- 172.21.15.188:0/767916093 shutdown_connections 2023-07-20T02:50:51.363 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.349+0000 7f92274ae700 1 -- 172.21.15.188:0/767916093 wait complete. 2023-07-20T02:50:51.363 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.349+0000 7f92274ae700 1 Processor -- start 2023-07-20T02:50:51.363 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.349+0000 7f92274ae700 1 -- start start 2023-07-20T02:50:51.364 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.349+0000 7f92274ae700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92200ff9a0 0x7f92201964e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:51.364 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.349+0000 7f92274ae700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9220100360 0x7f9220196a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:51.364 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.349+0000 7f92274ae700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9220100dd0 0x7f922019daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:51.364 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.349+0000 7f92274ae700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922019e070 con 0x7f92200ff9a0 2023-07-20T02:50:51.364 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.349+0000 7f92274ae700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922019e1b0 con 0x7f9220100dd0 2023-07-20T02:50:51.364 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.349+0000 7f92274ae700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922019e340 con 0x7f9220100360 2023-07-20T02:50:51.364 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.350+0000 7f9224a49700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9220100360 0x7f9220196a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:51.365 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.350+0000 7f9224a49700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9220100360 0x7f9220196a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:35338/0 (socket says 172.21.15.188:35338) 2023-07-20T02:50:51.365 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.350+0000 7f9224a49700 1 -- 172.21.15.188:0/2773564452 learned_addr learned my addr 172.21.15.188:0/2773564452 (peer_addr_for_me v2:172.21.15.188:0/0) 2023-07-20T02:50:51.365 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.350+0000 7f9224a49700 1 -- 172.21.15.188:0/2773564452 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9220100dd0 msgr2=0x7f922019daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:51.365 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.350+0000 7f9224a49700 1 --2- 172.21.15.188:0/2773564452 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9220100dd0 0x7f922019daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:51.365 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.350+0000 7f9224a49700 1 -- 172.21.15.188:0/2773564452 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92200ff9a0 msgr2=0x7f92201964e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:51.365 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.350+0000 7f9224a49700 1 --2- 172.21.15.188:0/2773564452 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92200ff9a0 0x7f92201964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:51.365 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.350+0000 7f9224a49700 1 -- 172.21.15.188:0/2773564452 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92100095f0 con 0x7f9220100360 2023-07-20T02:50:51.365 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.350+0000 7f9224a49700 1 --2- 172.21.15.188:0/2773564452 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9220100360 0x7f9220196a20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f921c00d980 tx=0x7f921c00dd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:50:51.366 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.350+0000 7f92167fc700 1 -- 172.21.15.188:0/2773564452 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921c009b40 con 0x7f9220100360 2023-07-20T02:50:51.367 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.350+0000 7f92274ae700 1 -- 172.21.15.188:0/2773564452 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f922019e620 con 0x7f9220100360 2023-07-20T02:50:51.367 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.350+0000 7f92167fc700 1 -- 172.21.15.188:0/2773564452 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f921c00b6e0 con 0x7f9220100360 2023-07-20T02:50:51.367 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.350+0000 7f92167fc700 1 -- 172.21.15.188:0/2773564452 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921c00f620 con 0x7f9220100360 2023-07-20T02:50:51.367 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.350+0000 7f92274ae700 1 -- 172.21.15.188:0/2773564452 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f922019eb20 con 0x7f9220100360 2023-07-20T02:50:51.367 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.351+0000 7f92274ae700 1 -- 172.21.15.188:0/2773564452 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9220106db0 con 0x7f9220100360 2023-07-20T02:50:51.367 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.352+0000 7f92167fc700 1 -- 172.21.15.188:0/2773564452 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f921c010460 con 0x7f9220100360 2023-07-20T02:50:51.367 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.353+0000 7f92167fc700 1 --2- 172.21.15.188:0/2773564452 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f920c0a1150 0x7f920c0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:51.368 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.353+0000 7f92167fc700 1 -- 172.21.15.188:0/2773564452 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f921c0cc3b0 con 0x7f9220100360 2023-07-20T02:50:51.368 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.353+0000 7f922524a700 1 --2- 172.21.15.188:0/2773564452 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f920c0a1150 0x7f920c0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:51.368 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.354+0000 7f922524a700 1 --2- 172.21.15.188:0/2773564452 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f920c0a1150 0x7f920c0a3610 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f92100095c0 tx=0x7f9210009530 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:50:51.368 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.357+0000 7f92167fc700 1 -- 172.21.15.188:0/2773564452 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f921c095fa0 con 0x7f9220100360 2023-07-20T02:50:51.563 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.551+0000 7f92274ae700 1 -- 172.21.15.188:0/2773564452 --> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi007=a;smithi007=d;smithi007=g;smithi007=j;smithi158=b;smithi158=e;smithi158=h;smithi158=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f9220060e70 con 0x7f920c0a1150 2023-07-20T02:50:51.572 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.559+0000 7f92167fc700 1 -- 172.21.15.188:0/2773564452 <== mgr.14152 v2:172.21.15.7:6800/1354542137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f9220060e70 con 0x7f920c0a1150 2023-07-20T02:50:51.572 INFO:teuthology.orchestra.run.smithi188.stdout:Scheduled mds.all update... 2023-07-20T02:50:51.574 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.562+0000 7f92274ae700 1 -- 172.21.15.188:0/2773564452 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f920c0a1150 msgr2=0x7f920c0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:51.575 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.562+0000 7f92274ae700 1 --2- 172.21.15.188:0/2773564452 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f920c0a1150 0x7f920c0a3610 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f92100095c0 tx=0x7f9210009530 comp rx=0 tx=0).stop 2023-07-20T02:50:51.575 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.562+0000 7f92274ae700 1 -- 172.21.15.188:0/2773564452 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9220100360 msgr2=0x7f9220196a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:51.575 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.562+0000 7f92274ae700 1 --2- 172.21.15.188:0/2773564452 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9220100360 0x7f9220196a20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f921c00d980 tx=0x7f921c00dd40 comp rx=0 tx=0).stop 2023-07-20T02:50:51.575 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.562+0000 7f92274ae700 1 -- 172.21.15.188:0/2773564452 shutdown_connections 2023-07-20T02:50:51.576 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.562+0000 7f92274ae700 1 --2- 172.21.15.188:0/2773564452 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9220100dd0 0x7f922019daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:51.576 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.562+0000 7f92274ae700 1 --2- 172.21.15.188:0/2773564452 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f920c0a1150 0x7f920c0a3610 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:51.576 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.562+0000 7f92274ae700 1 --2- 172.21.15.188:0/2773564452 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92200ff9a0 0x7f92201964e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:51.576 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.562+0000 7f92274ae700 1 --2- 172.21.15.188:0/2773564452 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9220100360 0x7f9220196a20 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:51.577 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.562+0000 7f92274ae700 1 -- 172.21.15.188:0/2773564452 >> 172.21.15.188:0/2773564452 conn(0x7f92200fa450 msgr2=0x7f9220103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:50:51.577 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.562+0000 7f92274ae700 1 -- 172.21.15.188:0/2773564452 shutdown_connections 2023-07-20T02:50:51.577 INFO:teuthology.orchestra.run.smithi188.stderr:2023-07-20T02:50:51.562+0000 7f92274ae700 1 -- 172.21.15.188:0/2773564452 wait complete. 2023-07-20T02:50:52.081 DEBUG:teuthology.orchestra.run.smithi007:mds.a> sudo journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.a.service 2023-07-20T02:50:52.086 DEBUG:teuthology.orchestra.run.smithi007:mds.d> sudo journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.d.service 2023-07-20T02:50:52.133 DEBUG:teuthology.orchestra.run.smithi007:mds.g> sudo journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.g.service 2023-07-20T02:50:52.169 INFO:journalctl@ceph.mds.a.smithi007.stdout:-- Logs begin at Thu 2023-07-20 02:22:32 UTC. -- 2023-07-20T02:50:52.175 DEBUG:teuthology.orchestra.run.smithi007:mds.j> sudo journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.j.service 2023-07-20T02:50:52.215 INFO:journalctl@ceph.mds.d.smithi007.stdout:-- Logs begin at Thu 2023-07-20 02:22:32 UTC. -- 2023-07-20T02:50:52.218 DEBUG:teuthology.orchestra.run.smithi158:mds.b> sudo journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.b.service 2023-07-20T02:50:52.220 DEBUG:teuthology.orchestra.run.smithi158:mds.e> sudo journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.e.service 2023-07-20T02:50:52.247 INFO:journalctl@ceph.mds.g.smithi007.stdout:-- Logs begin at Thu 2023-07-20 02:22:32 UTC. -- 2023-07-20T02:50:52.250 INFO:journalctl@ceph.mds.b.smithi158.stdout:-- Logs begin at Thu 2023-07-20 02:10:12 UTC. -- 2023-07-20T02:50:52.251 DEBUG:teuthology.orchestra.run.smithi158:mds.h> sudo journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.h.service 2023-07-20T02:50:52.278 INFO:journalctl@ceph.mds.e.smithi158.stdout:-- Logs begin at Thu 2023-07-20 02:10:12 UTC. -- 2023-07-20T02:50:52.279 DEBUG:teuthology.orchestra.run.smithi158:mds.k> sudo journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.k.service 2023-07-20T02:50:52.290 INFO:journalctl@ceph.mds.j.smithi007.stdout:-- Logs begin at Thu 2023-07-20 02:22:32 UTC. -- 2023-07-20T02:50:52.304 INFO:journalctl@ceph.mds.h.smithi158.stdout:-- Logs begin at Thu 2023-07-20 02:10:12 UTC. -- 2023-07-20T02:50:52.305 DEBUG:teuthology.orchestra.run.smithi188:mds.c> sudo journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.c.service 2023-07-20T02:50:52.308 DEBUG:teuthology.orchestra.run.smithi188:mds.f> sudo journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.f.service 2023-07-20T02:50:52.330 INFO:journalctl@ceph.mds.k.smithi158.stdout:-- Logs begin at Thu 2023-07-20 02:10:12 UTC. -- 2023-07-20T02:50:52.337 INFO:journalctl@ceph.mds.c.smithi188.stdout:-- Logs begin at Thu 2023-07-20 02:09:43 UTC. -- 2023-07-20T02:50:52.338 DEBUG:teuthology.orchestra.run.smithi188:mds.i> sudo journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.i.service 2023-07-20T02:50:52.365 INFO:journalctl@ceph.mds.f.smithi188.stdout:-- Logs begin at Thu 2023-07-20 02:09:43 UTC. -- 2023-07-20T02:50:52.366 DEBUG:teuthology.orchestra.run.smithi188:mds.l> sudo journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.l.service 2023-07-20T02:50:52.391 INFO:journalctl@ceph.mds.i.smithi188.stdout:-- Logs begin at Thu 2023-07-20 02:09:43 UTC. -- 2023-07-20T02:50:52.392 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2023-07-20T02:50:52.393 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T02:50:52.417 INFO:journalctl@ceph.mds.l.smithi188.stdout:-- Logs begin at Thu 2023-07-20 02:09:43 UTC. -- 2023-07-20T02:50:52.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.602+0000 7f3d2a9ff700 1 -- 172.21.15.7:0/2155804141 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d24100770 msgr2=0x7f3d24100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:52.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.602+0000 7f3d2a9ff700 1 --2- 172.21.15.7:0/2155804141 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d24100770 0x7f3d24100b90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3d140099c0 tx=0x7f3d14009cd0 comp rx=0 tx=0).stop 2023-07-20T02:50:52.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.602+0000 7f3d2a9ff700 1 -- 172.21.15.7:0/2155804141 shutdown_connections 2023-07-20T02:50:52.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.602+0000 7f3d2a9ff700 1 --2- 172.21.15.7:0/2155804141 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d24102300 0x7f3d24108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:52.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.602+0000 7f3d2a9ff700 1 --2- 172.21.15.7:0/2155804141 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d24100770 0x7f3d24100b90 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:52.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.602+0000 7f3d2a9ff700 1 --2- 172.21.15.7:0/2155804141 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3d24101960 0x7f3d24101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:52.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.602+0000 7f3d2a9ff700 1 -- 172.21.15.7:0/2155804141 >> 172.21.15.7:0/2155804141 conn(0x7f3d240fbca0 msgr2=0x7f3d240fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:50:52.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.603+0000 7f3d2a9ff700 1 -- 172.21.15.7:0/2155804141 shutdown_connections 2023-07-20T02:50:52.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.603+0000 7f3d2a9ff700 1 -- 172.21.15.7:0/2155804141 wait complete. 2023-07-20T02:50:52.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.604+0000 7f3d2a9ff700 1 Processor -- start 2023-07-20T02:50:52.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.604+0000 7f3d2a9ff700 1 -- start start 2023-07-20T02:50:52.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.605+0000 7f3d2a9ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3d24100770 0x7f3d24196440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:52.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.605+0000 7f3d2a9ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d24101960 0x7f3d24196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:52.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.605+0000 7f3d2a9ff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d24102300 0x7f3d2419d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:52.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.605+0000 7f3d2a9ff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2419dfc0 con 0x7f3d24101960 2023-07-20T02:50:52.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.605+0000 7f3d2a9ff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2419e100 con 0x7f3d24102300 2023-07-20T02:50:52.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.605+0000 7f3d2a9ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2419e290 con 0x7f3d24100770 2023-07-20T02:50:52.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.605+0000 7f3d23fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3d24100770 0x7f3d24196440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:52.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.605+0000 7f3d23fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3d24100770 0x7f3d24196440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.7:48532/0 (socket says 172.21.15.7:48532) 2023-07-20T02:50:52.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.605+0000 7f3d23fff700 1 -- 172.21.15.7:0/2706805626 learned_addr learned my addr 172.21.15.7:0/2706805626 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:50:52.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.605+0000 7f3d237fe700 1 --2- 172.21.15.7:0/2706805626 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d24101960 0x7f3d24196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:52.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.605+0000 7f3d28f9c700 1 --2- 172.21.15.7:0/2706805626 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d24102300 0x7f3d2419d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:52.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.605+0000 7f3d237fe700 1 -- 172.21.15.7:0/2706805626 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3d24100770 msgr2=0x7f3d24196440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:52.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.605+0000 7f3d237fe700 1 --2- 172.21.15.7:0/2706805626 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3d24100770 0x7f3d24196440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:52.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.605+0000 7f3d237fe700 1 -- 172.21.15.7:0/2706805626 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d24102300 msgr2=0x7f3d2419d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:52.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.605+0000 7f3d237fe700 1 --2- 172.21.15.7:0/2706805626 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d24102300 0x7f3d2419d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:52.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.605+0000 7f3d237fe700 1 -- 172.21.15.7:0/2706805626 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d140095f0 con 0x7f3d24101960 2023-07-20T02:50:52.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.605+0000 7f3d23fff700 1 --2- 172.21.15.7:0/2706805626 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3d24100770 0x7f3d24196440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T02:50:52.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.606+0000 7f3d237fe700 1 --2- 172.21.15.7:0/2706805626 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d24101960 0x7f3d24196980 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3d1800e9f0 tx=0x7f3d1800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:50:52.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.606+0000 7f3d217fa700 1 -- 172.21.15.7:0/2706805626 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1800c9c0 con 0x7f3d24101960 2023-07-20T02:50:52.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.606+0000 7f3d217fa700 1 -- 172.21.15.7:0/2706805626 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d18010430 con 0x7f3d24101960 2023-07-20T02:50:52.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.606+0000 7f3d2a9ff700 1 -- 172.21.15.7:0/2706805626 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d2419e570 con 0x7f3d24101960 2023-07-20T02:50:52.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.606+0000 7f3d217fa700 1 -- 172.21.15.7:0/2706805626 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d18018680 con 0x7f3d24101960 2023-07-20T02:50:52.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.606+0000 7f3d2a9ff700 1 -- 172.21.15.7:0/2706805626 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d2419ead0 con 0x7f3d24101960 2023-07-20T02:50:52.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.607+0000 7f3d2a9ff700 1 -- 172.21.15.7:0/2706805626 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d24107cb0 con 0x7f3d24101960 2023-07-20T02:50:52.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.609+0000 7f3d217fa700 1 -- 172.21.15.7:0/2706805626 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f3d18010860 con 0x7f3d24101960 2023-07-20T02:50:52.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.610+0000 7f3d217fa700 1 --2- 172.21.15.7:0/2706805626 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f3d0c0a1220 0x7f3d0c0a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:52.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.611+0000 7f3d217fa700 1 -- 172.21.15.7:0/2706805626 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3d18014070 con 0x7f3d24101960 2023-07-20T02:50:52.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.611+0000 7f3d23fff700 1 --2- 172.21.15.7:0/2706805626 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f3d0c0a1220 0x7f3d0c0a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:52.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.611+0000 7f3d23fff700 1 --2- 172.21.15.7:0/2706805626 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f3d0c0a1220 0x7f3d0c0a36e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3d14000c00 tx=0x7f3d14005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:50:52.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.615+0000 7f3d217fa700 1 -- 172.21.15.7:0/2706805626 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3d18051780 con 0x7f3d24101960 2023-07-20T02:50:52.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:52 smithi188 ceph-mon[129553]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:50:52.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:52 smithi188 ceph-mon[129553]: from='client.24448 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi007=a;smithi007=d;smithi007=g;smithi007=j;smithi158=b;smithi158=e;smithi158=h;smithi158=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:50:52.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:52 smithi188 ceph-mon[129553]: Saving service mds.all spec with placement smithi007=a;smithi007=d;smithi007=g;smithi007=j;smithi158=b;smithi158=e;smithi158=h;smithi158=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l;count:12 2023-07-20T02:50:52.714 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:52 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:52.714 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:52 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:50:52.714 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:52 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:50:52.714 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:52 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:50:52.714 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:52 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:52.714 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:52 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:50:52.714 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:52 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:50:52.714 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:52 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:50:52.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.827+0000 7f3d2a9ff700 1 -- 172.21.15.7:0/2706805626 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d2402c970 con 0x7f3d24101960 2023-07-20T02:50:52.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.828+0000 7f3d217fa700 1 -- 172.21.15.7:0/2706805626 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+18389 (secure 0 0 0) 0x7f3d18096120 con 0x7f3d24101960 2023-07-20T02:50:52.841 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:50:52.842 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":79,"fsid":"1b04643c-26a7-11ee-9b34-001a4aab830c","created":"2023-07-20T02:43:56.544719+0000","modified":"2023-07-20T02:50:41.133457+0000","last_up_change":"2023-07-20T02:50:40.126926+0000","last_in_change":"2023-07-20T02:50:21.875020+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T02:47:12.055539+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad6c6339-6e23-4c21-9fde-2dbecdafdb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6803","nonce":3322992091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6805","nonce":3322992091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6809","nonce":3322992091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6807","nonce":3322992091}]},"public_addr":"172.21.15.7:6803/3322992091","cluster_addr":"172.21.15.7:6805/3322992091","heartbeat_back_addr":"172.21.15.7:6809/3322992091","heartbeat_front_addr":"172.21.15.7:6807/3322992091","state":["exists","up"]},{"osd":1,"uuid":"cad207b3-a1ea-4f07-bd94-9b11fd573a69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6801","nonce":2796771612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6803","nonce":2796771612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6807","nonce":2796771612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6805","nonce":2796771612}]},"public_addr":"172.21.15.158:6801/2796771612","cluster_addr":"172.21.15.158:6803/2796771612","heartbeat_back_addr":"172.21.15.158:6807/2796771612","heartbeat_front_addr":"172.21.15.158:6805/2796771612","state":["exists","up"]},{"osd":2,"uuid":"2c777fe6-d003-49d0-af37-f63fca37ff06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6801","nonce":2928307258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6803","nonce":2928307258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6807","nonce":2928307258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6805","nonce":2928307258}]},"public_addr":"172.21.15.188:6801/2928307258","cluster_addr":"172.21.15.188:6803/2928307258","heartbeat_back_addr":"172.21.15.188:6807/2928307258","heartbeat_front_addr":"172.21.15.188:6805/2928307258","state":["exists","up"]},{"osd":3,"uuid":"59e29965-018f-4eb6-adbd-1a9e1df84245","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6811","nonce":2192953827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6813","nonce":2192953827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6817","nonce":2192953827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6815","nonce":2192953827}]},"public_addr":"172.21.15.7:6811/2192953827","cluster_addr":"172.21.15.7:6813/2192953827","heartbeat_back_addr":"172.21.15.7:6817/2192953827","heartbeat_front_addr":"172.21.15.7:6815/2192953827","state":["exists","up"]},{"osd":4,"uuid":"e2453384-3b09-4f2a-bea7-172d8fc466d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6809","nonce":3330591132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6810","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6811","nonce":3330591132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6814","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6815","nonce":3330591132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6813","nonce":3330591132}]},"public_addr":"172.21.15.158:6809/3330591132","cluster_addr":"172.21.15.158:6811/3330591132","heartbeat_back_addr":"172.21.15.158:6815/3330591132","heartbeat_front_addr":"172.21.15.158:6813/3330591132","state":["exists","up"]},{"osd":5,"uuid":"c3c04c13-0ce5-440d-9e62-0e301ad9f771","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6809","nonce":3142012805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6811","nonce":3142012805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6815","nonce":3142012805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6813","nonce":3142012805}]},"public_addr":"172.21.15.188:6809/3142012805","cluster_addr":"172.21.15.188:6811/3142012805","heartbeat_back_addr":"172.21.15.188:6815/3142012805","heartbeat_front_addr":"172.21.15.188:6813/3142012805","state":["exists","up"]},{"osd":6,"uuid":"82ef2d33-1045-41f1-bf69-b03404bcace8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6819","nonce":859560576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6821","nonce":859560576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6825","nonce":859560576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6823","nonce":859560576}]},"public_addr":"172.21.15.7:6819/859560576","cluster_addr":"172.21.15.7:6821/859560576","heartbeat_back_addr":"172.21.15.7:6825/859560576","heartbeat_front_addr":"172.21.15.7:6823/859560576","state":["exists","up"]},{"osd":7,"uuid":"020ec859-3fc2-4da2-a0dc-10530a4a366c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6817","nonce":2578758635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6819","nonce":2578758635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6822","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6823","nonce":2578758635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6821","nonce":2578758635}]},"public_addr":"172.21.15.158:6817/2578758635","cluster_addr":"172.21.15.158:6819/2578758635","heartbeat_back_addr":"172.21.15.158:6823/2578758635","heartbeat_front_addr":"172.21.15.158:6821/2578758635","state":["exists","up"]},{"osd":8,"uuid":"4081496a-45ee-4b25-8f17-a23665521187","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6817","nonce":3020824843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6819","nonce":3020824843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6823","nonce":3020824843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6821","nonce":3020824843}]},"public_addr":"172.21.15.188:6817/3020824843","cluster_addr":"172.21.15.188:6819/3020824843","heartbeat_back_addr":"172.21.15.188:6823/3020824843","heartbeat_front_addr":"172.21.15.188:6821/3020824843","state":["exists","up"]},{"osd":9,"uuid":"723580ae-efe3-4d86-95fd-d7a4c0631425","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6827","nonce":1612464825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6829","nonce":1612464825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6833","nonce":1612464825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6831","nonce":1612464825}]},"public_addr":"172.21.15.7:6827/1612464825","cluster_addr":"172.21.15.7:6829/1612464825","heartbeat_back_addr":"172.21.15.7:6833/1612464825","heartbeat_front_addr":"172.21.15.7:6831/1612464825","state":["exists","up"]},{"osd":10,"uuid":"c2651548-fe46-4d2d-b911-278036412ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6825","nonce":1653166210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6827","nonce":1653166210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6831","nonce":1653166210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6829","nonce":1653166210}]},"public_addr":"172.21.15.158:6825/1653166210","cluster_addr":"172.21.15.158:6827/1653166210","heartbeat_back_addr":"172.21.15.158:6831/1653166210","heartbeat_front_addr":"172.21.15.158:6829/1653166210","state":["exists","up"]},{"osd":11,"uuid":"f7262bcb-4243-403a-a321-fc2bdf1d4d25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6825","nonce":1230196525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6827","nonce":1230196525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6831","nonce":1230196525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6829","nonce":1230196525}]},"public_addr":"172.21.15.188:6825/1230196525","cluster_addr":"172.21.15.188:6827/1230196525","heartbeat_back_addr":"172.21.15.188:6831/1230196525","heartbeat_front_addr":"172.21.15.188:6829/1230196525","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:46:26.803832+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:46:48.726253+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:47:09.744123+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:47:36.907588+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:47:57.696685+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:48:19.120806+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:48:45.313043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:49:06.686384+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:49:28.475078+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:49:54.984033+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:50:16.541259+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:50:38.221746+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/3892248290":"2023-07-21T02:44:53.458649+0000","172.21.15.7:0/2373058722":"2023-07-21T02:44:53.458649+0000","172.21.15.7:6801/2926310425":"2023-07-21T02:44:53.458649+0000","172.21.15.7:0/4222569264":"2023-07-21T02:44:25.804258+0000","172.21.15.7:0/1357065986":"2023-07-21T02:44:25.804258+0000","172.21.15.7:0/884368977":"2023-07-21T02:44:53.458649+0000","172.21.15.7:6801/1224472946":"2023-07-21T02:44:25.804258+0000","172.21.15.7:0/1459293748":"2023-07-21T02:44:25.804258+0000","172.21.15.7:6800/1224472946":"2023-07-21T02:44:25.804258+0000","172.21.15.7:6800/2926310425":"2023-07-21T02:44:53.458649+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T02:50:52.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.830+0000 7f3d2a9ff700 1 -- 172.21.15.7:0/2706805626 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f3d0c0a1220 msgr2=0x7f3d0c0a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:52.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.830+0000 7f3d2a9ff700 1 --2- 172.21.15.7:0/2706805626 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f3d0c0a1220 0x7f3d0c0a36e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3d14000c00 tx=0x7f3d14005fb0 comp rx=0 tx=0).stop 2023-07-20T02:50:52.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.830+0000 7f3d2a9ff700 1 -- 172.21.15.7:0/2706805626 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d24101960 msgr2=0x7f3d24196980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:52.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.830+0000 7f3d2a9ff700 1 --2- 172.21.15.7:0/2706805626 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d24101960 0x7f3d24196980 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3d1800e9f0 tx=0x7f3d1800ed00 comp rx=0 tx=0).stop 2023-07-20T02:50:52.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.830+0000 7f3d2a9ff700 1 -- 172.21.15.7:0/2706805626 shutdown_connections 2023-07-20T02:50:52.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.830+0000 7f3d2a9ff700 1 --2- 172.21.15.7:0/2706805626 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3d24102300 0x7f3d2419d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:52.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.830+0000 7f3d2a9ff700 1 --2- 172.21.15.7:0/2706805626 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3d24100770 0x7f3d24196440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:52.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.830+0000 7f3d2a9ff700 1 --2- 172.21.15.7:0/2706805626 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f3d0c0a1220 0x7f3d0c0a36e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:52.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.830+0000 7f3d2a9ff700 1 --2- 172.21.15.7:0/2706805626 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d24101960 0x7f3d24196980 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:52.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.830+0000 7f3d2a9ff700 1 -- 172.21.15.7:0/2706805626 >> 172.21.15.7:0/2706805626 conn(0x7f3d240fbca0 msgr2=0x7f3d24104df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:50:52.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.830+0000 7f3d2a9ff700 1 -- 172.21.15.7:0/2706805626 shutdown_connections 2023-07-20T02:50:52.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:52.830+0000 7f3d2a9ff700 1 -- 172.21.15.7:0/2706805626 wait complete. 2023-07-20T02:50:52.861 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:52 smithi007 ceph-mon[126353]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:50:52.861 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:52 smithi007 ceph-mon[126353]: from='client.24448 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi007=a;smithi007=d;smithi007=g;smithi007=j;smithi158=b;smithi158=e;smithi158=h;smithi158=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:50:52.861 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:52 smithi007 ceph-mon[126353]: Saving service mds.all spec with placement smithi007=a;smithi007=d;smithi007=g;smithi007=j;smithi158=b;smithi158=e;smithi158=h;smithi158=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l;count:12 2023-07-20T02:50:52.861 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:52 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:52.861 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:52 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:50:52.861 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:52 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:50:52.862 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:52 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:50:52.862 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:52 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:52.862 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:52 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:50:52.862 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:52 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:50:52.862 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:52 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:50:52.863 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-20T02:47:12.055539+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-20T02:50:52.864 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-20T02:50:52.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:52 smithi158 ceph-mon[128925]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:50:52.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:52 smithi158 ceph-mon[128925]: from='client.24448 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi007=a;smithi007=d;smithi007=g;smithi007=j;smithi158=b;smithi158=e;smithi158=h;smithi158=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:50:52.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:52 smithi158 ceph-mon[128925]: Saving service mds.all spec with placement smithi007=a;smithi007=d;smithi007=g;smithi007=j;smithi158=b;smithi158=e;smithi158=h;smithi158=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l;count:12 2023-07-20T02:50:52.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:52 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:52.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:52 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:50:52.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:52 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:50:52.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:52 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:50:52.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:52 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:52.892 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:52 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:50:52.892 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:52 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:50:52.892 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:52 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:50:53.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.057+0000 7f844a3ff700 1 -- 172.21.15.7:0/2342463913 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8444073170 msgr2=0x7f84440735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:53.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.057+0000 7f844a3ff700 1 --2- 172.21.15.7:0/2342463913 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8444073170 0x7f84440735f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8438009990 tx=0x7f8438009ca0 comp rx=0 tx=0).stop 2023-07-20T02:50:53.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.057+0000 7f844a3ff700 1 -- 172.21.15.7:0/2342463913 shutdown_connections 2023-07-20T02:50:53.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.057+0000 7f844a3ff700 1 --2- 172.21.15.7:0/2342463913 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8444108630 0x7f844410aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:53.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.057+0000 7f844a3ff700 1 --2- 172.21.15.7:0/2342463913 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8444074750 0x7f8444072ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:53.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.057+0000 7f844a3ff700 1 --2- 172.21.15.7:0/2342463913 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8444073170 0x7f84440735f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:53.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.057+0000 7f844a3ff700 1 -- 172.21.15.7:0/2342463913 >> 172.21.15.7:0/2342463913 conn(0x7f84440fbcb0 msgr2=0x7f84440fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:50:53.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.058+0000 7f844a3ff700 1 -- 172.21.15.7:0/2342463913 shutdown_connections 2023-07-20T02:50:53.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.058+0000 7f844a3ff700 1 -- 172.21.15.7:0/2342463913 wait complete. 2023-07-20T02:50:53.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.059+0000 7f844a3ff700 1 Processor -- start 2023-07-20T02:50:53.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.059+0000 7f844a3ff700 1 -- start start 2023-07-20T02:50:53.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.059+0000 7f844a3ff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8444073170 0x7f84441986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:53.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.059+0000 7f844a3ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8444074750 0x7f8444198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:53.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.059+0000 7f844a3ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8444108630 0x7f844419fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:53.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.059+0000 7f844a3ff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84441a0220 con 0x7f8444108630 2023-07-20T02:50:53.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.059+0000 7f844a3ff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84441a0360 con 0x7f8444073170 2023-07-20T02:50:53.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.059+0000 7f844a3ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84441a04f0 con 0x7f8444074750 2023-07-20T02:50:53.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.059+0000 7f844899c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8444108630 0x7f844419fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:53.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.059+0000 7f84437fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8444074750 0x7f8444198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:53.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.059+0000 7f844899c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8444108630 0x7f844419fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35548/0 (socket says 172.21.15.7:35548) 2023-07-20T02:50:53.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.059+0000 7f844899c700 1 -- 172.21.15.7:0/3056655274 learned_addr learned my addr 172.21.15.7:0/3056655274 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:50:53.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.059+0000 7f8443fff700 1 --2- 172.21.15.7:0/3056655274 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8444073170 0x7f84441986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:53.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.060+0000 7f84437fe700 1 -- 172.21.15.7:0/3056655274 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8444073170 msgr2=0x7f84441986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:53.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.060+0000 7f84437fe700 1 --2- 172.21.15.7:0/3056655274 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8444073170 0x7f84441986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:53.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.060+0000 7f84437fe700 1 -- 172.21.15.7:0/3056655274 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8444108630 msgr2=0x7f844419fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:53.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.060+0000 7f84437fe700 1 --2- 172.21.15.7:0/3056655274 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8444108630 0x7f844419fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:53.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.060+0000 7f84437fe700 1 -- 172.21.15.7:0/3056655274 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84380095f0 con 0x7f8444074750 2023-07-20T02:50:53.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.060+0000 7f844899c700 1 --2- 172.21.15.7:0/3056655274 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8444108630 0x7f844419fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T02:50:53.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.060+0000 7f84437fe700 1 --2- 172.21.15.7:0/3056655274 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8444074750 0x7f8444198be0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f843800b680 tx=0x7f8438009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:50:53.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.060+0000 7f84417fa700 1 -- 172.21.15.7:0/3056655274 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843801d070 con 0x7f8444074750 2023-07-20T02:50:53.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.060+0000 7f844a3ff700 1 -- 172.21.15.7:0/3056655274 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84441a0770 con 0x7f8444074750 2023-07-20T02:50:53.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.060+0000 7f84417fa700 1 -- 172.21.15.7:0/3056655274 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8438021410 con 0x7f8444074750 2023-07-20T02:50:53.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.060+0000 7f84417fa700 1 -- 172.21.15.7:0/3056655274 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8438017650 con 0x7f8444074750 2023-07-20T02:50:53.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.060+0000 7f844a3ff700 1 -- 172.21.15.7:0/3056655274 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84441a0bf0 con 0x7f8444074750 2023-07-20T02:50:53.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.061+0000 7f844a3ff700 1 -- 172.21.15.7:0/3056655274 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8444106d30 con 0x7f8444074750 2023-07-20T02:50:53.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.063+0000 7f84417fa700 1 -- 172.21.15.7:0/3056655274 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f8438021810 con 0x7f8444074750 2023-07-20T02:50:53.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.065+0000 7f84417fa700 1 --2- 172.21.15.7:0/3056655274 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f842c0a1150 0x7f842c0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:53.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.065+0000 7f84417fa700 1 -- 172.21.15.7:0/3056655274 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f84380cd9f0 con 0x7f8444074750 2023-07-20T02:50:53.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.065+0000 7f8443fff700 1 --2- 172.21.15.7:0/3056655274 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f842c0a1150 0x7f842c0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:53.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.065+0000 7f8443fff700 1 --2- 172.21.15.7:0/3056655274 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f842c0a1150 0x7f842c0a3610 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8434009c60 tx=0x7f8434009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:50:53.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.067+0000 7f84417fa700 1 -- 172.21.15.7:0/3056655274 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f84380975d0 con 0x7f8444074750 2023-07-20T02:50:53.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.278+0000 7f844a3ff700 1 -- 172.21.15.7:0/3056655274 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f844402c970 con 0x7f8444074750 2023-07-20T02:50:53.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.278+0000 7f84417fa700 1 -- 172.21.15.7:0/3056655274 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7f843809abf0 con 0x7f8444074750 2023-07-20T02:50:53.291 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2023-07-20T02:50:53.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.281+0000 7f844a3ff700 1 -- 172.21.15.7:0/3056655274 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f842c0a1150 msgr2=0x7f842c0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:53.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.281+0000 7f844a3ff700 1 --2- 172.21.15.7:0/3056655274 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f842c0a1150 0x7f842c0a3610 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8434009c60 tx=0x7f8434009260 comp rx=0 tx=0).stop 2023-07-20T02:50:53.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.281+0000 7f844a3ff700 1 -- 172.21.15.7:0/3056655274 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8444074750 msgr2=0x7f8444198be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:53.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.281+0000 7f844a3ff700 1 --2- 172.21.15.7:0/3056655274 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8444074750 0x7f8444198be0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f843800b680 tx=0x7f8438009580 comp rx=0 tx=0).stop 2023-07-20T02:50:53.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.281+0000 7f844a3ff700 1 -- 172.21.15.7:0/3056655274 shutdown_connections 2023-07-20T02:50:53.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.281+0000 7f844a3ff700 1 --2- 172.21.15.7:0/3056655274 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f842c0a1150 0x7f842c0a3610 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:53.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.281+0000 7f844a3ff700 1 --2- 172.21.15.7:0/3056655274 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8444108630 0x7f844419fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:53.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.281+0000 7f844a3ff700 1 --2- 172.21.15.7:0/3056655274 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8444073170 0x7f84441986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:53.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.281+0000 7f844a3ff700 1 --2- 172.21.15.7:0/3056655274 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8444074750 0x7f8444198be0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:53.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.281+0000 7f844a3ff700 1 -- 172.21.15.7:0/3056655274 >> 172.21.15.7:0/3056655274 conn(0x7f84440fbcb0 msgr2=0x7f84440fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:50:53.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.281+0000 7f844a3ff700 1 -- 172.21.15.7:0/3056655274 shutdown_connections 2023-07-20T02:50:53.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.281+0000 7f844a3ff700 1 -- 172.21.15.7:0/3056655274 wait complete. 2023-07-20T02:50:53.313 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T02:50:53.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.504+0000 7fe5d4bff700 1 -- 172.21.15.7:0/4039765564 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5d0101960 msgr2=0x7fe5d0101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:53.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.504+0000 7fe5d4bff700 1 --2- 172.21.15.7:0/4039765564 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5d0101960 0x7fe5d0101dc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe5c4009990 tx=0x7fe5c4009ca0 comp rx=0 tx=0).stop 2023-07-20T02:50:53.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.504+0000 7fe5d4bff700 1 -- 172.21.15.7:0/4039765564 shutdown_connections 2023-07-20T02:50:53.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.504+0000 7fe5d4bff700 1 --2- 172.21.15.7:0/4039765564 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5d0102300 0x7fe5d0108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:53.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.504+0000 7fe5d4bff700 1 --2- 172.21.15.7:0/4039765564 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5d0100770 0x7fe5d0100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:53.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.504+0000 7fe5d4bff700 1 --2- 172.21.15.7:0/4039765564 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5d0101960 0x7fe5d0101dc0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:53.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.504+0000 7fe5d4bff700 1 -- 172.21.15.7:0/4039765564 >> 172.21.15.7:0/4039765564 conn(0x7fe5d00fbca0 msgr2=0x7fe5d00fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:50:53.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.504+0000 7fe5d4bff700 1 -- 172.21.15.7:0/4039765564 shutdown_connections 2023-07-20T02:50:53.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.505+0000 7fe5d4bff700 1 -- 172.21.15.7:0/4039765564 wait complete. 2023-07-20T02:50:53.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.506+0000 7fe5d4bff700 1 Processor -- start 2023-07-20T02:50:53.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.506+0000 7fe5d4bff700 1 -- start start 2023-07-20T02:50:53.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.507+0000 7fe5d4bff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5d0100770 0x7fe5d01963f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:53.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.507+0000 7fe5d4bff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5d0101960 0x7fe5d0196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:53.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.507+0000 7fe5d4bff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5d0102300 0x7fe5d019d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:53.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.507+0000 7fe5d4bff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d019df70 con 0x7fe5d0102300 2023-07-20T02:50:53.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.507+0000 7fe5d4bff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d019e0b0 con 0x7fe5d0100770 2023-07-20T02:50:53.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.507+0000 7fe5d4bff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d019e240 con 0x7fe5d0101960 2023-07-20T02:50:53.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.507+0000 7fe5ce59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5d0100770 0x7fe5d01963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:53.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.507+0000 7fe5cdd9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5d0101960 0x7fe5d0196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:53.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.507+0000 7fe5ce59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5d0100770 0x7fe5d01963f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.7:47848/0 (socket says 172.21.15.7:47848) 2023-07-20T02:50:53.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.507+0000 7fe5ce59c700 1 -- 172.21.15.7:0/3341911563 learned_addr learned my addr 172.21.15.7:0/3341911563 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:50:53.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.507+0000 7fe5ced9d700 1 --2- 172.21.15.7:0/3341911563 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5d0102300 0x7fe5d019d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:53.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.507+0000 7fe5ce59c700 1 -- 172.21.15.7:0/3341911563 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5d0101960 msgr2=0x7fe5d0196930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:53.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.507+0000 7fe5ce59c700 1 --2- 172.21.15.7:0/3341911563 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5d0101960 0x7fe5d0196930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:53.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.507+0000 7fe5ce59c700 1 -- 172.21.15.7:0/3341911563 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5d0102300 msgr2=0x7fe5d019d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:53.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.507+0000 7fe5ce59c700 1 --2- 172.21.15.7:0/3341911563 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5d0102300 0x7fe5d019d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:53.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.507+0000 7fe5ce59c700 1 -- 172.21.15.7:0/3341911563 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c0009520 con 0x7fe5d0100770 2023-07-20T02:50:53.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.508+0000 7fe5ce59c700 1 --2- 172.21.15.7:0/3341911563 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5d0100770 0x7fe5d01963f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe5c000ea50 tx=0x7fe5c000ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:50:53.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.508+0000 7fe5bf7fe700 1 -- 172.21.15.7:0/3341911563 <== mon.2 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c000cbd0 con 0x7fe5d0100770 2023-07-20T02:50:53.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.508+0000 7fe5bf7fe700 1 -- 172.21.15.7:0/3341911563 <== mon.2 v2:172.21.15.158:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5c000cd30 con 0x7fe5d0100770 2023-07-20T02:50:53.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.508+0000 7fe5d4bff700 1 -- 172.21.15.7:0/3341911563 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5c40095f0 con 0x7fe5d0100770 2023-07-20T02:50:53.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.508+0000 7fe5bf7fe700 1 -- 172.21.15.7:0/3341911563 <== mon.2 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c0018680 con 0x7fe5d0100770 2023-07-20T02:50:53.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.508+0000 7fe5d4bff700 1 -- 172.21.15.7:0/3341911563 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5d019e8a0 con 0x7fe5d0100770 2023-07-20T02:50:53.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.509+0000 7fe5d4bff700 1 -- 172.21.15.7:0/3341911563 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5d0107d10 con 0x7fe5d0100770 2023-07-20T02:50:53.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.512+0000 7fe5bf7fe700 1 -- 172.21.15.7:0/3341911563 <== mon.2 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fe5c0010430 con 0x7fe5d0100770 2023-07-20T02:50:53.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.513+0000 7fe5bf7fe700 1 --2- 172.21.15.7:0/3341911563 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fe5b80a11a0 0x7fe5b80a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:53.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.513+0000 7fe5bf7fe700 1 -- 172.21.15.7:0/3341911563 <== mon.2 v2:172.21.15.158:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe5c0014070 con 0x7fe5d0100770 2023-07-20T02:50:53.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.513+0000 7fe5cdd9b700 1 --2- 172.21.15.7:0/3341911563 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fe5b80a11a0 0x7fe5b80a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:53.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.514+0000 7fe5cdd9b700 1 --2- 172.21.15.7:0/3341911563 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fe5b80a11a0 0x7fe5b80a3660 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe5c400b2e0 tx=0x7fe5c4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:50:53.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.515+0000 7fe5bf7fe700 1 -- 172.21.15.7:0/3341911563 <== mon.2 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe5c0096b00 con 0x7fe5d0100770 2023-07-20T02:50:53.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.769+0000 7fe5d4bff700 1 -- 172.21.15.7:0/3341911563 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5d002cbe0 con 0x7fe5d0100770 2023-07-20T02:50:53.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.770+0000 7fe5bf7fe700 1 -- 172.21.15.7:0/3341911563 <== mon.2 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7fe5c009a120 con 0x7fe5d0100770 2023-07-20T02:50:53.783 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:50:53.783 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2023-07-20T02:50:53.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.773+0000 7fe5d4bff700 1 -- 172.21.15.7:0/3341911563 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fe5b80a11a0 msgr2=0x7fe5b80a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:53.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.773+0000 7fe5d4bff700 1 --2- 172.21.15.7:0/3341911563 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fe5b80a11a0 0x7fe5b80a3660 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe5c400b2e0 tx=0x7fe5c4005fb0 comp rx=0 tx=0).stop 2023-07-20T02:50:53.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.773+0000 7fe5d4bff700 1 -- 172.21.15.7:0/3341911563 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5d0100770 msgr2=0x7fe5d01963f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:53.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.773+0000 7fe5d4bff700 1 --2- 172.21.15.7:0/3341911563 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5d0100770 0x7fe5d01963f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe5c000ea50 tx=0x7fe5c000ed60 comp rx=0 tx=0).stop 2023-07-20T02:50:53.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.773+0000 7fe5d4bff700 1 -- 172.21.15.7:0/3341911563 shutdown_connections 2023-07-20T02:50:53.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.773+0000 7fe5d4bff700 1 --2- 172.21.15.7:0/3341911563 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fe5b80a11a0 0x7fe5b80a3660 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:53.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.773+0000 7fe5d4bff700 1 --2- 172.21.15.7:0/3341911563 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5d0102300 0x7fe5d019d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:53.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.773+0000 7fe5d4bff700 1 --2- 172.21.15.7:0/3341911563 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe5d0100770 0x7fe5d01963f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:53.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.773+0000 7fe5d4bff700 1 --2- 172.21.15.7:0/3341911563 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe5d0101960 0x7fe5d0196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:53.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.773+0000 7fe5d4bff700 1 -- 172.21.15.7:0/3341911563 >> 172.21.15.7:0/3341911563 conn(0x7fe5d00fbca0 msgr2=0x7fe5d0104e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:50:53.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.773+0000 7fe5d4bff700 1 -- 172.21.15.7:0/3341911563 shutdown_connections 2023-07-20T02:50:53.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:53.773+0000 7fe5d4bff700 1 -- 172.21.15.7:0/3341911563 wait complete. 2023-07-20T02:50:53.788 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 1 2023-07-20T02:50:53.835 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:53 smithi188 ceph-mon[129553]: Deploying daemon mds.l on smithi188 2023-07-20T02:50:53.835 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:53 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/2706805626' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:50:53.835 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:53 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/3056655274' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T02:50:53.836 INFO:journalctl@ceph.mds.l.smithi188.stdout:Jul 20 02:50:53 smithi188 systemd[1]: Starting Ceph mds.l for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:50:53.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:53 smithi158 ceph-mon[128925]: Deploying daemon mds.l on smithi188 2023-07-20T02:50:53.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:53 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/2706805626' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:50:53.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:53 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/3056655274' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T02:50:53.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:53 smithi007 ceph-mon[126353]: Deploying daemon mds.l on smithi188 2023-07-20T02:50:53.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:53 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/2706805626' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:50:53.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:53 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/3056655274' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T02:50:54.463 INFO:journalctl@ceph.mds.l.smithi188.stdout:Jul 20 02:50:54 smithi188 podman[154941]: 2023-07-20T02:50:54.713 INFO:journalctl@ceph.mds.l.smithi188.stdout:Jul 20 02:50:54 smithi188 bash[154941]: c1ed45acc8b785cfd0b325501d813ea708ffd7a3433f008d51013d493cfb65fe 2023-07-20T02:50:54.713 INFO:journalctl@ceph.mds.l.smithi188.stdout:Jul 20 02:50:54 smithi188 ceph-mds[154982]: -- 172.21.15.188:0/1321234074 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5602520771e0 con 0x56025204d000 2023-07-20T02:50:54.713 INFO:journalctl@ceph.mds.l.smithi188.stdout:Jul 20 02:50:54 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-l[154957]: starting mds.l at 2023-07-20T02:50:54.714 INFO:journalctl@ceph.mds.l.smithi188.stdout:Jul 20 02:50:54 smithi188 systemd[1]: Started Ceph mds.l for 1b04643c-26a7-11ee-9b34-001a4aab830c. 2023-07-20T02:50:54.714 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:54 smithi188 ceph-mon[129553]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:50:54.714 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:54 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/3341911563' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:50:54.714 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:54 smithi188 ceph-mon[129553]: mds.? [v2:172.21.15.188:6832/1435841351,v1:172.21.15.188:6833/1435841351] up:boot 2023-07-20T02:50:54.714 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:54 smithi188 ceph-mon[129553]: fsmap 1 up:standby 2023-07-20T02:50:54.714 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:54 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-20T02:50:54.715 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:54 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:54.715 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:54 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:54.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:54 smithi158 ceph-mon[128925]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:50:54.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:54 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/3341911563' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:50:54.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:54 smithi158 ceph-mon[128925]: mds.? [v2:172.21.15.188:6832/1435841351,v1:172.21.15.188:6833/1435841351] up:boot 2023-07-20T02:50:54.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:54 smithi158 ceph-mon[128925]: fsmap 1 up:standby 2023-07-20T02:50:54.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:54 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-20T02:50:54.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:54 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:54.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:54 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:54.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:54 smithi007 ceph-mon[126353]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:50:54.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:54 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/3341911563' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:50:54.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:54 smithi007 ceph-mon[126353]: mds.? [v2:172.21.15.188:6832/1435841351,v1:172.21.15.188:6833/1435841351] up:boot 2023-07-20T02:50:54.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:54 smithi007 ceph-mon[126353]: fsmap 1 up:standby 2023-07-20T02:50:54.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:54 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-20T02:50:54.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:54 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:54.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:54 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:55.806 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T02:50:55.869 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:55 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:55.869 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:55 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:50:55.869 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:55 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:50:55.869 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:55 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:50:55.870 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:55 smithi158 ceph-mon[128925]: Deploying daemon mds.k on smithi158 2023-07-20T02:50:55.874 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:55 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:55.875 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:55 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:50:55.875 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:55 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:50:55.875 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:55 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:50:55.875 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:55 smithi007 ceph-mon[126353]: Deploying daemon mds.k on smithi158 2023-07-20T02:50:55.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:55 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:55.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:55 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:50:55.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:55 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:50:55.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:55 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:50:55.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:55 smithi188 ceph-mon[129553]: Deploying daemon mds.k on smithi158 2023-07-20T02:50:56.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.009+0000 7f1e0e5ff700 1 -- 172.21.15.7:0/695877049 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e08102300 msgr2=0x7f1e08108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:56.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.009+0000 7f1e0e5ff700 1 --2- 172.21.15.7:0/695877049 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e08102300 0x7f1e08108bb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1e00009fc0 tx=0x7f1e0000b330 comp rx=0 tx=0).stop 2023-07-20T02:50:56.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.009+0000 7f1e0e5ff700 1 -- 172.21.15.7:0/695877049 shutdown_connections 2023-07-20T02:50:56.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.009+0000 7f1e0e5ff700 1 --2- 172.21.15.7:0/695877049 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e08102300 0x7f1e08108bb0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:56.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.009+0000 7f1e0e5ff700 1 --2- 172.21.15.7:0/695877049 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1e08100770 0x7f1e08100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:56.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.009+0000 7f1e0e5ff700 1 --2- 172.21.15.7:0/695877049 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e08101960 0x7f1e08101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:56.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.009+0000 7f1e0e5ff700 1 -- 172.21.15.7:0/695877049 >> 172.21.15.7:0/695877049 conn(0x7f1e080fbca0 msgr2=0x7f1e080fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:50:56.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.009+0000 7f1e0e5ff700 1 -- 172.21.15.7:0/695877049 shutdown_connections 2023-07-20T02:50:56.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.009+0000 7f1e0e5ff700 1 -- 172.21.15.7:0/695877049 wait complete. 2023-07-20T02:50:56.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.010+0000 7f1e0e5ff700 1 Processor -- start 2023-07-20T02:50:56.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.010+0000 7f1e0e5ff700 1 -- start start 2023-07-20T02:50:56.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.010+0000 7f1e0e5ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e08100770 0x7f1e081986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:56.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.010+0000 7f1e0e5ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e08101960 0x7f1e08198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:56.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.010+0000 7f1e0e5ff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1e08102300 0x7f1e0819fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:56.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.010+0000 7f1e0e5ff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e081a0240 con 0x7f1e08100770 2023-07-20T02:50:56.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.010+0000 7f1e0e5ff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e081a0380 con 0x7f1e08102300 2023-07-20T02:50:56.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.010+0000 7f1e0e5ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e081a0510 con 0x7f1e08101960 2023-07-20T02:50:56.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.011+0000 7f1e07fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e08100770 0x7f1e081986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:56.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.011+0000 7f1e07fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e08100770 0x7f1e081986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35574/0 (socket says 172.21.15.7:35574) 2023-07-20T02:50:56.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.011+0000 7f1e0cb9c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1e08102300 0x7f1e0819fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:56.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.011+0000 7f1e07fff700 1 -- 172.21.15.7:0/666635896 learned_addr learned my addr 172.21.15.7:0/666635896 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:50:56.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.011+0000 7f1e077fe700 1 --2- 172.21.15.7:0/666635896 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e08101960 0x7f1e08198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:56.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.011+0000 7f1e0cb9c700 1 -- 172.21.15.7:0/666635896 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e08101960 msgr2=0x7f1e08198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:56.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.011+0000 7f1e0cb9c700 1 --2- 172.21.15.7:0/666635896 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e08101960 0x7f1e08198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:56.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.011+0000 7f1e0cb9c700 1 -- 172.21.15.7:0/666635896 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e08100770 msgr2=0x7f1e081986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:56.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.011+0000 7f1e0cb9c700 1 --2- 172.21.15.7:0/666635896 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e08100770 0x7f1e081986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:56.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.011+0000 7f1e0cb9c700 1 -- 172.21.15.7:0/666635896 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df80095f0 con 0x7f1e08102300 2023-07-20T02:50:56.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.011+0000 7f1e07fff700 1 --2- 172.21.15.7:0/666635896 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e08100770 0x7f1e081986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T02:50:56.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.011+0000 7f1e0cb9c700 1 --2- 172.21.15.7:0/666635896 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1e08102300 0x7f1e0819fc70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1e0000b6c0 tx=0x7f1e00007800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:50:56.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.012+0000 7f1e057fa700 1 -- 172.21.15.7:0/666635896 <== mon.2 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0000e040 con 0x7f1e08102300 2023-07-20T02:50:56.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.012+0000 7f1e0e5ff700 1 -- 172.21.15.7:0/666635896 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e00009c40 con 0x7f1e08102300 2023-07-20T02:50:56.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.012+0000 7f1e057fa700 1 -- 172.21.15.7:0/666635896 <== mon.2 v2:172.21.15.158:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1e00004e20 con 0x7f1e08102300 2023-07-20T02:50:56.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.012+0000 7f1e057fa700 1 -- 172.21.15.7:0/666635896 <== mon.2 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e00005040 con 0x7f1e08102300 2023-07-20T02:50:56.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.012+0000 7f1e0e5ff700 1 -- 172.21.15.7:0/666635896 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e081a0a80 con 0x7f1e08102300 2023-07-20T02:50:56.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.013+0000 7f1e0e5ff700 1 -- 172.21.15.7:0/666635896 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e0810bf80 con 0x7f1e08102300 2023-07-20T02:50:56.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.015+0000 7f1e057fa700 1 -- 172.21.15.7:0/666635896 <== mon.2 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f1e0001d470 con 0x7f1e08102300 2023-07-20T02:50:56.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.016+0000 7f1e057fa700 1 --2- 172.21.15.7:0/666635896 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f1df00a11a0 0x7f1df00a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:56.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.016+0000 7f1e057fa700 1 -- 172.21.15.7:0/666635896 <== mon.2 v2:172.21.15.158:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f1e000cdab0 con 0x7f1e08102300 2023-07-20T02:50:56.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.016+0000 7f1e07fff700 1 --2- 172.21.15.7:0/666635896 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f1df00a11a0 0x7f1df00a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:56.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.017+0000 7f1e07fff700 1 --2- 172.21.15.7:0/666635896 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f1df00a11a0 0x7f1df00a3660 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1df8005950 tx=0x7f1df8009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:50:56.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.018+0000 7f1e057fa700 1 -- 172.21.15.7:0/666635896 <== mon.2 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1e00097690 con 0x7f1e08102300 2023-07-20T02:50:56.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.276+0000 7f1e0e5ff700 1 -- 172.21.15.7:0/666635896 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e0810c140 con 0x7f1e08102300 2023-07-20T02:50:56.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.277+0000 7f1e057fa700 1 -- 172.21.15.7:0/666635896 <== mon.2 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1260 (secure 0 0 0) 0x7f1e0009acb0 con 0x7f1e08102300 2023-07-20T02:50:56.292 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:50:56.292 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/1435841351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":1435841351},{"type":"v1","addr":"172.21.15.188:6833","nonce":1435841351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-07-20T02:50:56.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.280+0000 7f1e0e5ff700 1 -- 172.21.15.7:0/666635896 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f1df00a11a0 msgr2=0x7f1df00a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:56.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.280+0000 7f1e0e5ff700 1 --2- 172.21.15.7:0/666635896 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f1df00a11a0 0x7f1df00a3660 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1df8005950 tx=0x7f1df8009310 comp rx=0 tx=0).stop 2023-07-20T02:50:56.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.280+0000 7f1e0e5ff700 1 -- 172.21.15.7:0/666635896 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1e08102300 msgr2=0x7f1e0819fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:56.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.280+0000 7f1e0e5ff700 1 --2- 172.21.15.7:0/666635896 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1e08102300 0x7f1e0819fc70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1e0000b6c0 tx=0x7f1e00007800 comp rx=0 tx=0).stop 2023-07-20T02:50:56.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.280+0000 7f1e0e5ff700 1 -- 172.21.15.7:0/666635896 shutdown_connections 2023-07-20T02:50:56.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.280+0000 7f1e0e5ff700 1 --2- 172.21.15.7:0/666635896 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1e08102300 0x7f1e0819fc70 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:56.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.280+0000 7f1e0e5ff700 1 --2- 172.21.15.7:0/666635896 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f1df00a11a0 0x7f1df00a3660 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:56.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.280+0000 7f1e0e5ff700 1 --2- 172.21.15.7:0/666635896 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e08100770 0x7f1e081986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:56.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.280+0000 7f1e0e5ff700 1 --2- 172.21.15.7:0/666635896 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e08101960 0x7f1e08198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:56.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.280+0000 7f1e0e5ff700 1 -- 172.21.15.7:0/666635896 >> 172.21.15.7:0/666635896 conn(0x7f1e080fbca0 msgr2=0x7f1e081071a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:50:56.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.280+0000 7f1e0e5ff700 1 -- 172.21.15.7:0/666635896 shutdown_connections 2023-07-20T02:50:56.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:56.280+0000 7f1e0e5ff700 1 -- 172.21.15.7:0/666635896 wait complete. 2023-07-20T02:50:56.294 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 2 2023-07-20T02:50:56.508 INFO:journalctl@ceph.mds.k.smithi158.stdout:Jul 20 02:50:56 smithi158 systemd[1]: Starting Ceph mds.k for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:50:56.768 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:56 smithi158 ceph-mon[128925]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:50:56.769 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:56 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/666635896' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:50:56.769 INFO:journalctl@ceph.mds.k.smithi158.stdout:Jul 20 02:50:56 smithi158 podman[153174]: 2023-07-20T02:50:56.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:56 smithi007 ceph-mon[126353]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:50:56.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:56 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/666635896' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:50:56.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:56 smithi188 ceph-mon[129553]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:50:56.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:56 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/666635896' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:50:57.140 INFO:journalctl@ceph.mds.k.smithi158.stdout:Jul 20 02:50:56 smithi158 ceph-mds[153214]: -- 172.21.15.158:0/300465197 <== mon.2 v2:172.21.15.158:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56141411b1e0 con 0x5614140f0c00 2023-07-20T02:50:57.140 INFO:journalctl@ceph.mds.k.smithi158.stdout:Jul 20 02:50:56 smithi158 bash[153174]: 62dfb9b221dc9409bedfefef01e494759237f461586940d0d5171d1f0eba893d 2023-07-20T02:50:57.141 INFO:journalctl@ceph.mds.k.smithi158.stdout:Jul 20 02:50:56 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-k[153190]: starting mds.k at 2023-07-20T02:50:57.141 INFO:journalctl@ceph.mds.k.smithi158.stdout:Jul 20 02:50:56 smithi158 systemd[1]: Started Ceph mds.k for 1b04643c-26a7-11ee-9b34-001a4aab830c. 2023-07-20T02:50:58.050 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:57 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:58.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:57 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:58.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:57 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:58.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:57 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:50:58.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:57 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:50:58.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:57 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:50:58.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:57 smithi007 ceph-mon[126353]: Deploying daemon mds.j on smithi007 2023-07-20T02:50:58.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:57 smithi007 ceph-mon[126353]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:50:58.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:57 smithi007 ceph-mon[126353]: mds.? [v2:172.21.15.158:6832/401861265,v1:172.21.15.158:6833/401861265] up:boot 2023-07-20T02:50:58.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:57 smithi007 ceph-mon[126353]: fsmap 2 up:standby 2023-07-20T02:50:58.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:57 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-20T02:50:58.140 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:57 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:58.141 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:57 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:58.141 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:57 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:58.141 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:57 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:50:58.141 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:57 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:50:58.141 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:57 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:50:58.142 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:57 smithi158 ceph-mon[128925]: Deploying daemon mds.j on smithi007 2023-07-20T02:50:58.142 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:57 smithi158 ceph-mon[128925]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:50:58.142 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:57 smithi158 ceph-mon[128925]: mds.? [v2:172.21.15.158:6832/401861265,v1:172.21.15.158:6833/401861265] up:boot 2023-07-20T02:50:58.142 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:57 smithi158 ceph-mon[128925]: fsmap 2 up:standby 2023-07-20T02:50:58.142 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:57 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-20T02:50:58.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:57 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:58.213 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:57 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:58.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:57 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:50:58.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:57 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:50:58.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:57 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:50:58.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:57 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:50:58.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:57 smithi188 ceph-mon[129553]: Deploying daemon mds.j on smithi007 2023-07-20T02:50:58.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:57 smithi188 ceph-mon[129553]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:50:58.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:57 smithi188 ceph-mon[129553]: mds.? [v2:172.21.15.158:6832/401861265,v1:172.21.15.158:6833/401861265] up:boot 2023-07-20T02:50:58.215 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:57 smithi188 ceph-mon[129553]: fsmap 2 up:standby 2023-07-20T02:50:58.215 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:57 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-20T02:50:58.313 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T02:50:58.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.523+0000 7ff46c1ff700 1 -- 172.21.15.7:0/2983306310 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff464101970 msgr2=0x7ff464101dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:58.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.523+0000 7ff46c1ff700 1 --2- 172.21.15.7:0/2983306310 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff464101970 0x7ff464101dd0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff454009920 tx=0x7ff454009c30 comp rx=0 tx=0).stop 2023-07-20T02:50:58.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.523+0000 7ff46c1ff700 1 -- 172.21.15.7:0/2983306310 shutdown_connections 2023-07-20T02:50:58.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.523+0000 7ff46c1ff700 1 --2- 172.21.15.7:0/2983306310 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff464102310 0x7ff464108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:58.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.523+0000 7ff46c1ff700 1 --2- 172.21.15.7:0/2983306310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff464100780 0x7ff464100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:58.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.523+0000 7ff46c1ff700 1 --2- 172.21.15.7:0/2983306310 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff464101970 0x7ff464101dd0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:58.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.523+0000 7ff46c1ff700 1 -- 172.21.15.7:0/2983306310 >> 172.21.15.7:0/2983306310 conn(0x7ff4640fbcb0 msgr2=0x7ff4640fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:50:58.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.523+0000 7ff46c1ff700 1 -- 172.21.15.7:0/2983306310 shutdown_connections 2023-07-20T02:50:58.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.523+0000 7ff46c1ff700 1 -- 172.21.15.7:0/2983306310 wait complete. 2023-07-20T02:50:58.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.525+0000 7ff46c1ff700 1 Processor -- start 2023-07-20T02:50:58.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.525+0000 7ff46c1ff700 1 -- start start 2023-07-20T02:50:58.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.525+0000 7ff46c1ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff464100780 0x7ff464198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:58.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.525+0000 7ff46c1ff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff464101970 0x7ff464198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:58.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.525+0000 7ff46c1ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff464102310 0x7ff46419fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:58.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.525+0000 7ff46c1ff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4641a0290 con 0x7ff464102310 2023-07-20T02:50:58.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.525+0000 7ff46c1ff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4641a03d0 con 0x7ff464101970 2023-07-20T02:50:58.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.525+0000 7ff46c1ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4641a0560 con 0x7ff464100780 2023-07-20T02:50:58.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.525+0000 7ff46a79c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff464102310 0x7ff46419fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:58.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.525+0000 7ff46a79c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff464102310 0x7ff46419fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35590/0 (socket says 172.21.15.7:35590) 2023-07-20T02:50:58.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.525+0000 7ff46a79c700 1 -- 172.21.15.7:0/134894257 learned_addr learned my addr 172.21.15.7:0/134894257 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:50:58.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.525+0000 7ff469f9b700 1 --2- 172.21.15.7:0/134894257 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff464100780 0x7ff464198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:58.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.525+0000 7ff46979a700 1 --2- 172.21.15.7:0/134894257 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff464101970 0x7ff464198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:58.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.526+0000 7ff46a79c700 1 -- 172.21.15.7:0/134894257 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff464100780 msgr2=0x7ff464198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:58.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.526+0000 7ff46a79c700 1 --2- 172.21.15.7:0/134894257 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff464100780 0x7ff464198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:58.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.526+0000 7ff46a79c700 1 -- 172.21.15.7:0/134894257 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff464101970 msgr2=0x7ff464198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:58.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.526+0000 7ff46a79c700 1 --2- 172.21.15.7:0/134894257 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff464101970 0x7ff464198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:58.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.526+0000 7ff46a79c700 1 -- 172.21.15.7:0/134894257 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff454009520 con 0x7ff464102310 2023-07-20T02:50:58.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.526+0000 7ff46a79c700 1 --2- 172.21.15.7:0/134894257 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff464102310 0x7ff46419fcc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff45c00ef30 tx=0x7ff45c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:50:58.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.526+0000 7ff45affd700 1 -- 172.21.15.7:0/134894257 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff45c018070 con 0x7ff464102310 2023-07-20T02:50:58.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.526+0000 7ff45affd700 1 -- 172.21.15.7:0/134894257 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff45c00f040 con 0x7ff464102310 2023-07-20T02:50:58.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.526+0000 7ff46c1ff700 1 -- 172.21.15.7:0/134894257 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4641a07e0 con 0x7ff464102310 2023-07-20T02:50:58.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.526+0000 7ff45affd700 1 -- 172.21.15.7:0/134894257 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff45c00ccf0 con 0x7ff464102310 2023-07-20T02:50:58.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.526+0000 7ff46c1ff700 1 -- 172.21.15.7:0/134894257 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4641a0ca0 con 0x7ff464102310 2023-07-20T02:50:58.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.528+0000 7ff46c1ff700 1 -- 172.21.15.7:0/134894257 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff46410bff0 con 0x7ff464102310 2023-07-20T02:50:58.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.530+0000 7ff45affd700 1 -- 172.21.15.7:0/134894257 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7ff45c003a00 con 0x7ff464102310 2023-07-20T02:50:58.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.532+0000 7ff45affd700 1 --2- 172.21.15.7:0/134894257 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7ff4500a11a0 0x7ff4500a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:50:58.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.532+0000 7ff45affd700 1 -- 172.21.15.7:0/134894257 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff45c0cc060 con 0x7ff464102310 2023-07-20T02:50:58.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.532+0000 7ff469f9b700 1 --2- 172.21.15.7:0/134894257 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7ff4500a11a0 0x7ff4500a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:50:58.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.533+0000 7ff469f9b700 1 --2- 172.21.15.7:0/134894257 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7ff4500a11a0 0x7ff4500a3660 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff4641017d0 tx=0x7ff460009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:50:58.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.536+0000 7ff45affd700 1 -- 172.21.15.7:0/134894257 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff45c095c40 con 0x7ff464102310 2023-07-20T02:50:58.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.804+0000 7ff46c1ff700 1 -- 172.21.15.7:0/134894257 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff46410c210 con 0x7ff464102310 2023-07-20T02:50:58.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.805+0000 7ff45affd700 1 -- 172.21.15.7:0/134894257 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2017 (secure 0 0 0) 0x7ff45c099260 con 0x7ff464102310 2023-07-20T02:50:58.818 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:50:58.818 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/1435841351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":1435841351},{"type":"v1","addr":"172.21.15.188:6833","nonce":1435841351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/401861265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":401861265},{"type":"v1","addr":"172.21.15.158:6833","nonce":401861265}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-07-20T02:50:58.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.808+0000 7ff46c1ff700 1 -- 172.21.15.7:0/134894257 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7ff4500a11a0 msgr2=0x7ff4500a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:58.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.808+0000 7ff46c1ff700 1 --2- 172.21.15.7:0/134894257 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7ff4500a11a0 0x7ff4500a3660 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff4641017d0 tx=0x7ff460009310 comp rx=0 tx=0).stop 2023-07-20T02:50:58.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.808+0000 7ff46c1ff700 1 -- 172.21.15.7:0/134894257 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff464102310 msgr2=0x7ff46419fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:50:58.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.808+0000 7ff46c1ff700 1 --2- 172.21.15.7:0/134894257 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff464102310 0x7ff46419fcc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff45c00ef30 tx=0x7ff45c00c3a0 comp rx=0 tx=0).stop 2023-07-20T02:50:58.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.809+0000 7ff46c1ff700 1 -- 172.21.15.7:0/134894257 shutdown_connections 2023-07-20T02:50:58.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.809+0000 7ff46c1ff700 1 --2- 172.21.15.7:0/134894257 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7ff4500a11a0 0x7ff4500a3660 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:58.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.809+0000 7ff46c1ff700 1 --2- 172.21.15.7:0/134894257 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff464102310 0x7ff46419fcc0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:58.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.809+0000 7ff46c1ff700 1 --2- 172.21.15.7:0/134894257 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff464100780 0x7ff464198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:58.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.809+0000 7ff46c1ff700 1 --2- 172.21.15.7:0/134894257 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff464101970 0x7ff464198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:50:58.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.809+0000 7ff46c1ff700 1 -- 172.21.15.7:0/134894257 >> 172.21.15.7:0/134894257 conn(0x7ff4640fbcb0 msgr2=0x7ff4640fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:50:58.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.810+0000 7ff46c1ff700 1 -- 172.21.15.7:0/134894257 shutdown_connections 2023-07-20T02:50:58.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:50:58.810+0000 7ff46c1ff700 1 -- 172.21.15.7:0/134894257 wait complete. 2023-07-20T02:50:58.826 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 3 2023-07-20T02:50:59.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:50:59 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/134894257' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:50:59.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:50:59 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/134894257' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:50:59.449 INFO:journalctl@ceph.mds.j.smithi007.stdout:Jul 20 02:50:59 smithi007 systemd[1]: Starting Ceph mds.j for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:50:59.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:50:59 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/134894257' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:50:59.807 INFO:journalctl@ceph.mds.j.smithi007.stdout:Jul 20 02:50:59 smithi007 podman[159978]: 2023-07-20T02:50:59.807 INFO:journalctl@ceph.mds.j.smithi007.stdout:Jul 20 02:50:59 smithi007 bash[159978]: 1b0587b7d5a049d53541f085ba509c88ec76bb176179f4e7a7e5ef57a5f629f2 2023-07-20T02:51:00.143 INFO:journalctl@ceph.mds.j.smithi007.stdout:Jul 20 02:50:59 smithi007 ceph-mds[160018]: -- 172.21.15.7:0/1668983353 <== mon.0 v2:172.21.15.7:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556742a6f1e0 con 0x556742a45000 2023-07-20T02:51:00.144 INFO:journalctl@ceph.mds.j.smithi007.stdout:Jul 20 02:50:59 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-j[159994]: starting mds.j at 2023-07-20T02:51:00.144 INFO:journalctl@ceph.mds.j.smithi007.stdout:Jul 20 02:50:59 smithi007 systemd[1]: Started Ceph mds.j for 1b04643c-26a7-11ee-9b34-001a4aab830c. 2023-07-20T02:51:00.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:00 smithi007 ceph-mon[126353]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:00.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:00 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:00.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:00 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:00.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:00 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:00.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:00 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:51:00.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:00 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:51:00.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:00 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:51:00.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:00 smithi188 ceph-mon[129553]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:00.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:00 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:00.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:00 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:00.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:00 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:00.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:00 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:51:00.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:00 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:51:00.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:00 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:51:00.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:00 smithi158 ceph-mon[128925]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:00.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:00 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:00.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:00 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:00.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:00 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:00.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:00 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:51:00.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:00 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:51:00.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:00 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:51:00.854 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T02:51:01.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.067+0000 7fefbf39d700 1 -- 172.21.15.7:0/1442572111 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefb80747b0 msgr2=0x7fefb8072c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:01.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.067+0000 7fefbf39d700 1 --2- 172.21.15.7:0/1442572111 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefb80747b0 0x7fefb8072c00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fefb40098f0 tx=0x7fefb4009c00 comp rx=0 tx=0).stop 2023-07-20T02:51:01.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.067+0000 7fefbf39d700 1 -- 172.21.15.7:0/1442572111 shutdown_connections 2023-07-20T02:51:01.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.067+0000 7fefbf39d700 1 --2- 172.21.15.7:0/1442572111 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fefb810cbb0 0x7fefb810efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:01.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.067+0000 7fefbf39d700 1 --2- 172.21.15.7:0/1442572111 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefb80747b0 0x7fefb8072c00 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:01.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.067+0000 7fefbf39d700 1 --2- 172.21.15.7:0/1442572111 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fefb80731d0 0x7fefb8073650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:01.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.067+0000 7fefbf39d700 1 -- 172.21.15.7:0/1442572111 >> 172.21.15.7:0/1442572111 conn(0x7fefb80fbb70 msgr2=0x7fefb80fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:01.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.067+0000 7fefbf39d700 1 -- 172.21.15.7:0/1442572111 shutdown_connections 2023-07-20T02:51:01.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.068+0000 7fefbf39d700 1 -- 172.21.15.7:0/1442572111 wait complete. 2023-07-20T02:51:01.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.069+0000 7fefbf39d700 1 Processor -- start 2023-07-20T02:51:01.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.069+0000 7fefbf39d700 1 -- start start 2023-07-20T02:51:01.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.070+0000 7fefbf39d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fefb80731d0 0x7fefb819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:01.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.070+0000 7fefbf39d700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fefb80747b0 0x7fefb819cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:01.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.070+0000 7fefbf39d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefb810cbb0 0x7fefb81a3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:01.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.070+0000 7fefbf39d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb81a43a0 con 0x7fefb80731d0 2023-07-20T02:51:01.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.070+0000 7fefbf39d700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb81a44e0 con 0x7fefb80747b0 2023-07-20T02:51:01.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.070+0000 7fefbf39d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb81a4670 con 0x7fefb810cbb0 2023-07-20T02:51:01.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.070+0000 7fefbe39b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fefb80731d0 0x7fefb819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:01.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.070+0000 7fefbe39b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fefb80731d0 0x7fefb819c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35616/0 (socket says 172.21.15.7:35616) 2023-07-20T02:51:01.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.070+0000 7fefbe39b700 1 -- 172.21.15.7:0/37878944 learned_addr learned my addr 172.21.15.7:0/37878944 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:01.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.070+0000 7fefbeb9c700 1 --2- 172.21.15.7:0/37878944 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefb810cbb0 0x7fefb81a3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:01.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.070+0000 7fefbdb9a700 1 --2- 172.21.15.7:0/37878944 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fefb80747b0 0x7fefb819cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:01.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.070+0000 7fefbe39b700 1 -- 172.21.15.7:0/37878944 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefb810cbb0 msgr2=0x7fefb81a3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:01.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.070+0000 7fefbe39b700 1 --2- 172.21.15.7:0/37878944 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefb810cbb0 0x7fefb81a3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:01.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.071+0000 7fefbe39b700 1 -- 172.21.15.7:0/37878944 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fefb80747b0 msgr2=0x7fefb819cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:01.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.071+0000 7fefbe39b700 1 --2- 172.21.15.7:0/37878944 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fefb80747b0 0x7fefb819cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:01.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.071+0000 7fefbe39b700 1 -- 172.21.15.7:0/37878944 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb0009d00 con 0x7fefb80731d0 2023-07-20T02:51:01.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.071+0000 7fefbeb9c700 1 --2- 172.21.15.7:0/37878944 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefb810cbb0 0x7fefb81a3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T02:51:01.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.071+0000 7fefbe39b700 1 --2- 172.21.15.7:0/37878944 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fefb80731d0 0x7fefb819c820 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fefb4009fd0 tx=0x7fefb400b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:01.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.071+0000 7fefaf7fe700 1 -- 172.21.15.7:0/37878944 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb401c070 con 0x7fefb80731d0 2023-07-20T02:51:01.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.071+0000 7fefaf7fe700 1 -- 172.21.15.7:0/37878944 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fefb400f7d0 con 0x7fefb80731d0 2023-07-20T02:51:01.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.071+0000 7fefaf7fe700 1 -- 172.21.15.7:0/37878944 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb4004ad0 con 0x7fefb80731d0 2023-07-20T02:51:01.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.071+0000 7fefbf39d700 1 -- 172.21.15.7:0/37878944 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefb4009520 con 0x7fefb80731d0 2023-07-20T02:51:01.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.071+0000 7fefbf39d700 1 -- 172.21.15.7:0/37878944 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefb81a4b80 con 0x7fefb80731d0 2023-07-20T02:51:01.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.073+0000 7fefbf39d700 1 -- 172.21.15.7:0/37878944 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefb8102700 con 0x7fefb80731d0 2023-07-20T02:51:01.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.074+0000 7fefaf7fe700 1 -- 172.21.15.7:0/37878944 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fefb4020070 con 0x7fefb80731d0 2023-07-20T02:51:01.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.076+0000 7fefaf7fe700 1 --2- 172.21.15.7:0/37878944 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fefa40a12c0 0x7fefa40a3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:01.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.076+0000 7fefaf7fe700 1 -- 172.21.15.7:0/37878944 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fefb40cd0a0 con 0x7fefb80731d0 2023-07-20T02:51:01.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.076+0000 7fefbdb9a700 1 --2- 172.21.15.7:0/37878944 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fefa40a12c0 0x7fefa40a3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:01.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.077+0000 7fefbdb9a700 1 --2- 172.21.15.7:0/37878944 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fefa40a12c0 0x7fefa40a3780 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fefb80744d0 tx=0x7fefa8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:01.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.080+0000 7fefaf7fe700 1 -- 172.21.15.7:0/37878944 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fefb405c780 con 0x7fefb80731d0 2023-07-20T02:51:01.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.347+0000 7fefbf39d700 1 -- 172.21.15.7:0/37878944 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fefb802cdc0 con 0x7fefb80731d0 2023-07-20T02:51:01.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.347+0000 7fefaf7fe700 1 -- 172.21.15.7:0/37878944 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2771 (secure 0 0 0) 0x7fefb4096c80 con 0x7fefb80731d0 2023-07-20T02:51:01.361 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:51:01.361 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/1435841351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":1435841351},{"type":"v1","addr":"172.21.15.188:6833","nonce":1435841351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/401861265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":401861265},{"type":"v1","addr":"172.21.15.158:6833","nonce":401861265}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/4064058311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":4064058311},{"type":"v1","addr":"172.21.15.7:6835","nonce":4064058311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4}],"filesystems":[]} 2023-07-20T02:51:01.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.350+0000 7fefbf39d700 1 -- 172.21.15.7:0/37878944 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fefa40a12c0 msgr2=0x7fefa40a3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:01.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.350+0000 7fefbf39d700 1 --2- 172.21.15.7:0/37878944 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fefa40a12c0 0x7fefa40a3780 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fefb80744d0 tx=0x7fefa8009260 comp rx=0 tx=0).stop 2023-07-20T02:51:01.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.350+0000 7fefbf39d700 1 -- 172.21.15.7:0/37878944 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fefb80731d0 msgr2=0x7fefb819c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:01.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.350+0000 7fefbf39d700 1 --2- 172.21.15.7:0/37878944 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fefb80731d0 0x7fefb819c820 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fefb4009fd0 tx=0x7fefb400b790 comp rx=0 tx=0).stop 2023-07-20T02:51:01.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.350+0000 7fefbf39d700 1 -- 172.21.15.7:0/37878944 shutdown_connections 2023-07-20T02:51:01.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.350+0000 7fefbf39d700 1 --2- 172.21.15.7:0/37878944 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefb810cbb0 0x7fefb81a3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:01.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.350+0000 7fefbf39d700 1 --2- 172.21.15.7:0/37878944 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fefa40a12c0 0x7fefa40a3780 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:01.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.350+0000 7fefbf39d700 1 --2- 172.21.15.7:0/37878944 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fefb80731d0 0x7fefb819c820 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:01.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.350+0000 7fefbf39d700 1 --2- 172.21.15.7:0/37878944 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fefb80747b0 0x7fefb819cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:01.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.350+0000 7fefbf39d700 1 -- 172.21.15.7:0/37878944 >> 172.21.15.7:0/37878944 conn(0x7fefb80fbb70 msgr2=0x7fefb80fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:01.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.350+0000 7fefbf39d700 1 -- 172.21.15.7:0/37878944 shutdown_connections 2023-07-20T02:51:01.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:01.350+0000 7fefbf39d700 1 -- 172.21.15.7:0/37878944 wait complete. 2023-07-20T02:51:01.365 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 4 2023-07-20T02:51:01.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:01 smithi007 ceph-mon[126353]: Deploying daemon mds.d on smithi007 2023-07-20T02:51:01.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:01 smithi007 ceph-mon[126353]: mds.? [v2:172.21.15.7:6834/4064058311,v1:172.21.15.7:6835/4064058311] up:boot 2023-07-20T02:51:01.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:01 smithi007 ceph-mon[126353]: fsmap 3 up:standby 2023-07-20T02:51:01.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:01 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-20T02:51:01.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:01 smithi188 ceph-mon[129553]: Deploying daemon mds.d on smithi007 2023-07-20T02:51:01.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:01 smithi188 ceph-mon[129553]: mds.? [v2:172.21.15.7:6834/4064058311,v1:172.21.15.7:6835/4064058311] up:boot 2023-07-20T02:51:01.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:01 smithi188 ceph-mon[129553]: fsmap 3 up:standby 2023-07-20T02:51:01.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:01 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-20T02:51:01.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:01 smithi158 ceph-mon[128925]: Deploying daemon mds.d on smithi007 2023-07-20T02:51:01.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:01 smithi158 ceph-mon[128925]: mds.? [v2:172.21.15.7:6834/4064058311,v1:172.21.15.7:6835/4064058311] up:boot 2023-07-20T02:51:01.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:01 smithi158 ceph-mon[128925]: fsmap 3 up:standby 2023-07-20T02:51:01.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:01 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-20T02:51:02.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:02 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/37878944' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:02.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:02 smithi007 ceph-mon[126353]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:02.318 INFO:journalctl@ceph.mds.d.smithi007.stdout:Jul 20 02:51:02 smithi007 systemd[1]: Starting Ceph mds.d for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:51:02.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:02 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/37878944' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:02.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:02 smithi188 ceph-mon[129553]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:02.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:02 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/37878944' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:02.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:02 smithi158 ceph-mon[128925]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:03.054 INFO:journalctl@ceph.mds.d.smithi007.stdout:Jul 20 02:51:02 smithi007 podman[160369]: 2023-07-20T02:51:03.383 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T02:51:03.448 INFO:journalctl@ceph.mds.d.smithi007.stdout:Jul 20 02:51:03 smithi007 bash[160369]: a3c801b742540222e19181946318b7db5a5287cf6d92ae2c1e74d52476e8a6d0 2023-07-20T02:51:03.448 INFO:journalctl@ceph.mds.d.smithi007.stdout:Jul 20 02:51:03 smithi007 ceph-mds[160410]: -- 172.21.15.7:0/3448696924 <== mon.0 v2:172.21.15.7:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f324b771e0 con 0x55f324b4d000 2023-07-20T02:51:03.448 INFO:journalctl@ceph.mds.d.smithi007.stdout:Jul 20 02:51:03 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-d[160385]: starting mds.d at 2023-07-20T02:51:03.448 INFO:journalctl@ceph.mds.d.smithi007.stdout:Jul 20 02:51:03 smithi007 systemd[1]: Started Ceph mds.d for 1b04643c-26a7-11ee-9b34-001a4aab830c. 2023-07-20T02:51:03.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.580+0000 7fc775bff700 1 -- 172.21.15.7:0/1457169725 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc770101960 msgr2=0x7fc770101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:03.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.580+0000 7fc775bff700 1 --2- 172.21.15.7:0/1457169725 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc770101960 0x7fc770101dc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc7640098f0 tx=0x7fc764009c00 comp rx=0 tx=0).stop 2023-07-20T02:51:03.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.580+0000 7fc775bff700 1 -- 172.21.15.7:0/1457169725 shutdown_connections 2023-07-20T02:51:03.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.580+0000 7fc775bff700 1 --2- 172.21.15.7:0/1457169725 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc770102300 0x7fc770108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:03.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.580+0000 7fc775bff700 1 --2- 172.21.15.7:0/1457169725 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc770100770 0x7fc770100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:03.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.580+0000 7fc775bff700 1 --2- 172.21.15.7:0/1457169725 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc770101960 0x7fc770101dc0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:03.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.580+0000 7fc775bff700 1 -- 172.21.15.7:0/1457169725 >> 172.21.15.7:0/1457169725 conn(0x7fc7700fbca0 msgr2=0x7fc7700fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:03.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.580+0000 7fc775bff700 1 -- 172.21.15.7:0/1457169725 shutdown_connections 2023-07-20T02:51:03.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.580+0000 7fc775bff700 1 -- 172.21.15.7:0/1457169725 wait complete. 2023-07-20T02:51:03.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.582+0000 7fc775bff700 1 Processor -- start 2023-07-20T02:51:03.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.582+0000 7fc775bff700 1 -- start start 2023-07-20T02:51:03.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.582+0000 7fc775bff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc770100770 0x7fc770196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:03.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.582+0000 7fc775bff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc770101960 0x7fc770196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:03.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.583+0000 7fc775bff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc770102300 0x7fc77019da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:03.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.583+0000 7fc775bff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77019dfd0 con 0x7fc770100770 2023-07-20T02:51:03.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.583+0000 7fc775bff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77019e110 con 0x7fc770102300 2023-07-20T02:51:03.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.583+0000 7fc775bff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77019e2a0 con 0x7fc770101960 2023-07-20T02:51:03.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.583+0000 7fc76f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc770100770 0x7fc770196450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:03.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.583+0000 7fc76f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc770100770 0x7fc770196450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36170/0 (socket says 172.21.15.7:36170) 2023-07-20T02:51:03.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.583+0000 7fc76f7fe700 1 -- 172.21.15.7:0/1660963398 learned_addr learned my addr 172.21.15.7:0/1660963398 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:03.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.583+0000 7fc76effd700 1 --2- 172.21.15.7:0/1660963398 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc770101960 0x7fc770196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:03.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.583+0000 7fc76ffff700 1 --2- 172.21.15.7:0/1660963398 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc770102300 0x7fc77019da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:03.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.583+0000 7fc76f7fe700 1 -- 172.21.15.7:0/1660963398 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc770101960 msgr2=0x7fc770196990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:03.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.583+0000 7fc76f7fe700 1 --2- 172.21.15.7:0/1660963398 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc770101960 0x7fc770196990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:03.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.583+0000 7fc76f7fe700 1 -- 172.21.15.7:0/1660963398 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc770102300 msgr2=0x7fc77019da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:03.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.583+0000 7fc76f7fe700 1 --2- 172.21.15.7:0/1660963398 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc770102300 0x7fc77019da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:03.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.583+0000 7fc76f7fe700 1 -- 172.21.15.7:0/1660963398 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc764009520 con 0x7fc770100770 2023-07-20T02:51:03.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.583+0000 7fc76effd700 1 --2- 172.21.15.7:0/1660963398 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc770101960 0x7fc770196990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T02:51:03.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.584+0000 7fc76f7fe700 1 --2- 172.21.15.7:0/1660963398 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc770100770 0x7fc770196450 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc76000e970 tx=0x7fc76000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:03.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.584+0000 7fc76cff9700 1 -- 172.21.15.7:0/1660963398 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc760009790 con 0x7fc770100770 2023-07-20T02:51:03.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.584+0000 7fc76cff9700 1 -- 172.21.15.7:0/1660963398 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc76000c7e0 con 0x7fc770100770 2023-07-20T02:51:03.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.584+0000 7fc775bff700 1 -- 172.21.15.7:0/1660963398 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc77019e580 con 0x7fc770100770 2023-07-20T02:51:03.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.584+0000 7fc76cff9700 1 -- 172.21.15.7:0/1660963398 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc760003c70 con 0x7fc770100770 2023-07-20T02:51:03.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.584+0000 7fc775bff700 1 -- 172.21.15.7:0/1660963398 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc77019ea90 con 0x7fc770100770 2023-07-20T02:51:03.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.585+0000 7fc775bff700 1 -- 172.21.15.7:0/1660963398 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7701076f0 con 0x7fc770100770 2023-07-20T02:51:03.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.587+0000 7fc76cff9700 1 -- 172.21.15.7:0/1660963398 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fc76000cc50 con 0x7fc770100770 2023-07-20T02:51:03.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.590+0000 7fc76cff9700 1 --2- 172.21.15.7:0/1660963398 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fc7580a11a0 0x7fc7580a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:03.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.590+0000 7fc76cff9700 1 -- 172.21.15.7:0/1660963398 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc760014070 con 0x7fc770100770 2023-07-20T02:51:03.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.590+0000 7fc76effd700 1 --2- 172.21.15.7:0/1660963398 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fc7580a11a0 0x7fc7580a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:03.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.590+0000 7fc76effd700 1 --2- 172.21.15.7:0/1660963398 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fc7580a11a0 0x7fc7580a3660 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc76400fa70 tx=0x7fc764019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:03.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.592+0000 7fc76cff9700 1 -- 172.21.15.7:0/1660963398 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc760096370 con 0x7fc770100770 2023-07-20T02:51:03.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.847+0000 7fc775bff700 1 -- 172.21.15.7:0/1660963398 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc77002cc20 con 0x7fc770100770 2023-07-20T02:51:03.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.847+0000 7fc76cff9700 1 -- 172.21.15.7:0/1660963398 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3522 (secure 0 0 0) 0x7fc760099990 con 0x7fc770100770 2023-07-20T02:51:03.861 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:51:03.861 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/190105946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":190105946},{"type":"v1","addr":"172.21.15.7:6837","nonce":190105946}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/1435841351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":1435841351},{"type":"v1","addr":"172.21.15.188:6833","nonce":1435841351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/401861265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":401861265},{"type":"v1","addr":"172.21.15.158:6833","nonce":401861265}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/4064058311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":4064058311},{"type":"v1","addr":"172.21.15.7:6835","nonce":4064058311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4}],"filesystems":[]} 2023-07-20T02:51:03.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.850+0000 7fc775bff700 1 -- 172.21.15.7:0/1660963398 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fc7580a11a0 msgr2=0x7fc7580a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:03.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.850+0000 7fc775bff700 1 --2- 172.21.15.7:0/1660963398 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fc7580a11a0 0x7fc7580a3660 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc76400fa70 tx=0x7fc764019040 comp rx=0 tx=0).stop 2023-07-20T02:51:03.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.850+0000 7fc775bff700 1 -- 172.21.15.7:0/1660963398 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc770100770 msgr2=0x7fc770196450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:03.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.850+0000 7fc775bff700 1 --2- 172.21.15.7:0/1660963398 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc770100770 0x7fc770196450 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc76000e970 tx=0x7fc76000ec80 comp rx=0 tx=0).stop 2023-07-20T02:51:03.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.850+0000 7fc775bff700 1 -- 172.21.15.7:0/1660963398 shutdown_connections 2023-07-20T02:51:03.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.850+0000 7fc775bff700 1 --2- 172.21.15.7:0/1660963398 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc770102300 0x7fc77019da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:03.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.850+0000 7fc775bff700 1 --2- 172.21.15.7:0/1660963398 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fc7580a11a0 0x7fc7580a3660 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:03.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.850+0000 7fc775bff700 1 --2- 172.21.15.7:0/1660963398 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc770100770 0x7fc770196450 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:03.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.850+0000 7fc775bff700 1 --2- 172.21.15.7:0/1660963398 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc770101960 0x7fc770196990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:03.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.850+0000 7fc775bff700 1 -- 172.21.15.7:0/1660963398 >> 172.21.15.7:0/1660963398 conn(0x7fc7700fbca0 msgr2=0x7fc7700fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:03.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.850+0000 7fc775bff700 1 -- 172.21.15.7:0/1660963398 shutdown_connections 2023-07-20T02:51:03.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:03.850+0000 7fc775bff700 1 -- 172.21.15.7:0/1660963398 wait complete. 2023-07-20T02:51:03.865 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 5 2023-07-20T02:51:04.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:04 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:04.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:04 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:04.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:04 smithi158 ceph-mon[128925]: mds.? [v2:172.21.15.7:6836/190105946,v1:172.21.15.7:6837/190105946] up:boot 2023-07-20T02:51:04.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:04 smithi158 ceph-mon[128925]: fsmap 4 up:standby 2023-07-20T02:51:04.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:04 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-20T02:51:04.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:04 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:04.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:04 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:51:04.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:04 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:51:04.392 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:04 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:51:04.392 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:04 smithi158 ceph-mon[128925]: Deploying daemon mds.e on smithi158 2023-07-20T02:51:04.392 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:04 smithi158 ceph-mon[128925]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:04.392 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:04 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/1660963398' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:04.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:04 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:04.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:04 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:04.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:04 smithi007 ceph-mon[126353]: mds.? [v2:172.21.15.7:6836/190105946,v1:172.21.15.7:6837/190105946] up:boot 2023-07-20T02:51:04.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:04 smithi007 ceph-mon[126353]: fsmap 4 up:standby 2023-07-20T02:51:04.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:04 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-20T02:51:04.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:04 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:04.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:04 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:51:04.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:04 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:51:04.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:04 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:51:04.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:04 smithi007 ceph-mon[126353]: Deploying daemon mds.e on smithi158 2023-07-20T02:51:04.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:04 smithi007 ceph-mon[126353]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:04.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:04 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/1660963398' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:04.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:04 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:04.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:04 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:04.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:04 smithi188 ceph-mon[129553]: mds.? [v2:172.21.15.7:6836/190105946,v1:172.21.15.7:6837/190105946] up:boot 2023-07-20T02:51:04.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:04 smithi188 ceph-mon[129553]: fsmap 4 up:standby 2023-07-20T02:51:04.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:04 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-20T02:51:04.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:04 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:04.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:04 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:51:04.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:04 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:51:04.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:04 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:51:04.466 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:04 smithi188 ceph-mon[129553]: Deploying daemon mds.e on smithi158 2023-07-20T02:51:04.466 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:04 smithi188 ceph-mon[129553]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:04.466 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:04 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/1660963398' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:04.891 INFO:journalctl@ceph.mds.e.smithi158.stdout:Jul 20 02:51:04 smithi158 systemd[1]: Starting Ceph mds.e for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:51:05.265 INFO:journalctl@ceph.mds.e.smithi158.stdout:Jul 20 02:51:04 smithi158 podman[153517]: 2023-07-20T02:51:05.266 INFO:journalctl@ceph.mds.e.smithi158.stdout:Jul 20 02:51:05 smithi158 ceph-mds[153557]: -- 172.21.15.158:0/1211454742 <== mon.2 v2:172.21.15.158:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a35b10d1e0 con 0x55a35b0e3000 2023-07-20T02:51:05.266 INFO:journalctl@ceph.mds.e.smithi158.stdout:Jul 20 02:51:05 smithi158 bash[153517]: b3f771ac733eba426b84bae382f5b2e823b47f45ff3bfb4c01cae8a3f7f125e5 2023-07-20T02:51:05.266 INFO:journalctl@ceph.mds.e.smithi158.stdout:Jul 20 02:51:05 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-e[153533]: starting mds.e at 2023-07-20T02:51:05.640 INFO:journalctl@ceph.mds.e.smithi158.stdout:Jul 20 02:51:05 smithi158 systemd[1]: Started Ceph mds.e for 1b04643c-26a7-11ee-9b34-001a4aab830c. 2023-07-20T02:51:05.882 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T02:51:06.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.085+0000 7f24f23ff700 1 -- 172.21.15.7:0/516088794 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24ec101960 msgr2=0x7f24ec101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:06.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.085+0000 7f24f23ff700 1 --2- 172.21.15.7:0/516088794 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24ec101960 0x7f24ec101dc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f24e00099c0 tx=0x7f24e0009cd0 comp rx=0 tx=0).stop 2023-07-20T02:51:06.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.085+0000 7f24f23ff700 1 -- 172.21.15.7:0/516088794 shutdown_connections 2023-07-20T02:51:06.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.085+0000 7f24f23ff700 1 --2- 172.21.15.7:0/516088794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24ec102300 0x7f24ec108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:06.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.085+0000 7f24f23ff700 1 --2- 172.21.15.7:0/516088794 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f24ec100770 0x7f24ec100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:06.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.085+0000 7f24f23ff700 1 --2- 172.21.15.7:0/516088794 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24ec101960 0x7f24ec101dc0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:06.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.085+0000 7f24f23ff700 1 -- 172.21.15.7:0/516088794 >> 172.21.15.7:0/516088794 conn(0x7f24ec0fbca0 msgr2=0x7f24ec0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:06.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.085+0000 7f24f23ff700 1 -- 172.21.15.7:0/516088794 shutdown_connections 2023-07-20T02:51:06.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.085+0000 7f24f23ff700 1 -- 172.21.15.7:0/516088794 wait complete. 2023-07-20T02:51:06.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.086+0000 7f24f23ff700 1 Processor -- start 2023-07-20T02:51:06.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.086+0000 7f24f23ff700 1 -- start start 2023-07-20T02:51:06.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.086+0000 7f24f23ff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f24ec100770 0x7f24ec198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:06.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.086+0000 7f24f23ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24ec101960 0x7f24ec198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:06.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.086+0000 7f24f23ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24ec102300 0x7f24ec19fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:06.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.086+0000 7f24f23ff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24ec1a01d0 con 0x7f24ec102300 2023-07-20T02:51:06.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.086+0000 7f24f23ff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24ec1a0310 con 0x7f24ec100770 2023-07-20T02:51:06.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.086+0000 7f24f23ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24ec1a04a0 con 0x7f24ec101960 2023-07-20T02:51:06.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.087+0000 7f24ebfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f24ec100770 0x7f24ec198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:06.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.087+0000 7f24ebfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f24ec100770 0x7f24ec198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.7:34636/0 (socket says 172.21.15.7:34636) 2023-07-20T02:51:06.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.087+0000 7f24ebfff700 1 -- 172.21.15.7:0/4209938735 learned_addr learned my addr 172.21.15.7:0/4209938735 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:06.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.087+0000 7f24f099c700 1 --2- 172.21.15.7:0/4209938735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24ec102300 0x7f24ec19fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:06.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.087+0000 7f24eb7fe700 1 --2- 172.21.15.7:0/4209938735 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24ec101960 0x7f24ec198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:06.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.087+0000 7f24ebfff700 1 -- 172.21.15.7:0/4209938735 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24ec101960 msgr2=0x7f24ec198b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:06.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.087+0000 7f24ebfff700 1 --2- 172.21.15.7:0/4209938735 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24ec101960 0x7f24ec198b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:06.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.087+0000 7f24ebfff700 1 -- 172.21.15.7:0/4209938735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24ec102300 msgr2=0x7f24ec19fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:06.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.087+0000 7f24ebfff700 1 --2- 172.21.15.7:0/4209938735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24ec102300 0x7f24ec19fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:06.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.087+0000 7f24ebfff700 1 -- 172.21.15.7:0/4209938735 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e00095f0 con 0x7f24ec100770 2023-07-20T02:51:06.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.087+0000 7f24eb7fe700 1 --2- 172.21.15.7:0/4209938735 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24ec101960 0x7f24ec198b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T02:51:06.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.087+0000 7f24ebfff700 1 --2- 172.21.15.7:0/4209938735 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f24ec100770 0x7f24ec198650 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f24dc00e970 tx=0x7f24dc00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:06.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.088+0000 7f24e97fa700 1 -- 172.21.15.7:0/4209938735 <== mon.2 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24dc009790 con 0x7f24ec100770 2023-07-20T02:51:06.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.088+0000 7f24f23ff700 1 -- 172.21.15.7:0/4209938735 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24ec1a0780 con 0x7f24ec100770 2023-07-20T02:51:06.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.088+0000 7f24e97fa700 1 -- 172.21.15.7:0/4209938735 <== mon.2 v2:172.21.15.158:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f24dc00c7e0 con 0x7f24ec100770 2023-07-20T02:51:06.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.088+0000 7f24e97fa700 1 -- 172.21.15.7:0/4209938735 <== mon.2 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24dc003b60 con 0x7f24ec100770 2023-07-20T02:51:06.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.088+0000 7f24f23ff700 1 -- 172.21.15.7:0/4209938735 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24ec1a0ce0 con 0x7f24ec100770 2023-07-20T02:51:06.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.089+0000 7f24f23ff700 1 -- 172.21.15.7:0/4209938735 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24ec10bfe0 con 0x7f24ec100770 2023-07-20T02:51:06.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.091+0000 7f24e97fa700 1 -- 172.21.15.7:0/4209938735 <== mon.2 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f24dc003d40 con 0x7f24ec100770 2023-07-20T02:51:06.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.092+0000 7f24e97fa700 1 --2- 172.21.15.7:0/4209938735 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f24d40a1220 0x7f24d40a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:06.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.093+0000 7f24e97fa700 1 -- 172.21.15.7:0/4209938735 <== mon.2 v2:172.21.15.158:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f24dc014070 con 0x7f24ec100770 2023-07-20T02:51:06.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.093+0000 7f24eb7fe700 1 --2- 172.21.15.7:0/4209938735 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f24d40a1220 0x7f24d40a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:06.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.093+0000 7f24eb7fe700 1 --2- 172.21.15.7:0/4209938735 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f24d40a1220 0x7f24d40a36e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f24e000b580 tx=0x7f24e0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:06.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.094+0000 7f24e97fa700 1 -- 172.21.15.7:0/4209938735 <== mon.2 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f24dc0967a0 con 0x7f24ec100770 2023-07-20T02:51:06.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.347+0000 7f24f23ff700 1 -- 172.21.15.7:0/4209938735 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24ec1a1380 con 0x7f24ec100770 2023-07-20T02:51:06.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.350+0000 7f24e97fa700 1 -- 172.21.15.7:0/4209938735 <== mon.2 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4282 (secure 0 0 0) 0x7f24dc099dc0 con 0x7f24ec100770 2023-07-20T02:51:06.363 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:51:06.364 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/190105946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":190105946},{"type":"v1","addr":"172.21.15.7:6837","nonce":190105946}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/1435841351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":1435841351},{"type":"v1","addr":"172.21.15.188:6833","nonce":1435841351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/401861265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":401861265},{"type":"v1","addr":"172.21.15.158:6833","nonce":401861265}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/4064058311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":4064058311},{"type":"v1","addr":"172.21.15.7:6835","nonce":4064058311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3931487788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3931487788},{"type":"v1","addr":"172.21.15.158:6835","nonce":3931487788}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2023-07-20T02:51:06.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.353+0000 7f24f23ff700 1 -- 172.21.15.7:0/4209938735 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f24d40a1220 msgr2=0x7f24d40a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:06.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.353+0000 7f24f23ff700 1 --2- 172.21.15.7:0/4209938735 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f24d40a1220 0x7f24d40a36e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f24e000b580 tx=0x7f24e0005fb0 comp rx=0 tx=0).stop 2023-07-20T02:51:06.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.353+0000 7f24f23ff700 1 -- 172.21.15.7:0/4209938735 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f24ec100770 msgr2=0x7f24ec198650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:06.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.353+0000 7f24f23ff700 1 --2- 172.21.15.7:0/4209938735 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f24ec100770 0x7f24ec198650 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f24dc00e970 tx=0x7f24dc00ec80 comp rx=0 tx=0).stop 2023-07-20T02:51:06.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.353+0000 7f24f23ff700 1 -- 172.21.15.7:0/4209938735 shutdown_connections 2023-07-20T02:51:06.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.353+0000 7f24f23ff700 1 --2- 172.21.15.7:0/4209938735 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f24d40a1220 0x7f24d40a36e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:06.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.353+0000 7f24f23ff700 1 --2- 172.21.15.7:0/4209938735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24ec102300 0x7f24ec19fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:06.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.353+0000 7f24f23ff700 1 --2- 172.21.15.7:0/4209938735 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f24ec100770 0x7f24ec198650 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:06.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.353+0000 7f24f23ff700 1 --2- 172.21.15.7:0/4209938735 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24ec101960 0x7f24ec198b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:06.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.353+0000 7f24f23ff700 1 -- 172.21.15.7:0/4209938735 >> 172.21.15.7:0/4209938735 conn(0x7f24ec0fbca0 msgr2=0x7f24ec0fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:06.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.353+0000 7f24f23ff700 1 -- 172.21.15.7:0/4209938735 shutdown_connections 2023-07-20T02:51:06.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:06.353+0000 7f24f23ff700 1 -- 172.21.15.7:0/4209938735 wait complete. 2023-07-20T02:51:06.368 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 6 2023-07-20T02:51:06.603 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:06 smithi188 ceph-mon[129553]: mds.? [v2:172.21.15.158:6834/3931487788,v1:172.21.15.158:6835/3931487788] up:boot 2023-07-20T02:51:06.603 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:06 smithi188 ceph-mon[129553]: fsmap 5 up:standby 2023-07-20T02:51:06.604 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:06 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-20T02:51:06.604 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:06 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:06.604 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:06 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:06.604 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:06 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:06.604 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:06 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:51:06.604 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:06 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:51:06.604 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:06 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:51:06.605 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:06 smithi188 ceph-mon[129553]: Deploying daemon mds.f on smithi188 2023-07-20T02:51:06.605 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:06 smithi188 ceph-mon[129553]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:06.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:06 smithi158 ceph-mon[128925]: mds.? [v2:172.21.15.158:6834/3931487788,v1:172.21.15.158:6835/3931487788] up:boot 2023-07-20T02:51:06.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:06 smithi158 ceph-mon[128925]: fsmap 5 up:standby 2023-07-20T02:51:06.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:06 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-20T02:51:06.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:06 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:06.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:06 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:06.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:06 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:06.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:06 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:51:06.642 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:06 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:51:06.642 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:06 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:51:06.642 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:06 smithi158 ceph-mon[128925]: Deploying daemon mds.f on smithi188 2023-07-20T02:51:06.642 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:06 smithi158 ceph-mon[128925]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:06.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:06 smithi007 ceph-mon[126353]: mds.? [v2:172.21.15.158:6834/3931487788,v1:172.21.15.158:6835/3931487788] up:boot 2023-07-20T02:51:06.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:06 smithi007 ceph-mon[126353]: fsmap 5 up:standby 2023-07-20T02:51:06.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:06 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-20T02:51:06.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:06 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:06.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:06 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:06.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:06 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:06.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:06 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:51:06.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:06 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:51:06.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:06 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:51:06.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:06 smithi007 ceph-mon[126353]: Deploying daemon mds.f on smithi188 2023-07-20T02:51:06.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:06 smithi007 ceph-mon[126353]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:06.939 INFO:journalctl@ceph.mds.f.smithi188.stdout:Jul 20 02:51:06 smithi188 systemd[1]: Starting Ceph mds.f for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:51:07.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:07 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/4209938735' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:07.464 INFO:journalctl@ceph.mds.f.smithi188.stdout:Jul 20 02:51:07 smithi188 podman[155286]: 2023-07-20T02:51:07.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:07 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/4209938735' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:07.700 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:07 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/4209938735' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:07.963 INFO:journalctl@ceph.mds.f.smithi188.stdout:Jul 20 02:51:07 smithi188 ceph-mds[155326]: -- 172.21.15.188:0/639366271 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55982c0851e0 con 0x55982c05b000 2023-07-20T02:51:07.964 INFO:journalctl@ceph.mds.f.smithi188.stdout:Jul 20 02:51:07 smithi188 bash[155286]: dc1a82c18a75d6d0442365f34af44fafbe0e301a824d934b214f75f9cb11e238 2023-07-20T02:51:07.964 INFO:journalctl@ceph.mds.f.smithi188.stdout:Jul 20 02:51:07 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-f[155302]: starting mds.f at 2023-07-20T02:51:07.964 INFO:journalctl@ceph.mds.f.smithi188.stdout:Jul 20 02:51:07 smithi188 systemd[1]: Started Ceph mds.f for 1b04643c-26a7-11ee-9b34-001a4aab830c. 2023-07-20T02:51:08.386 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T02:51:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.587+0000 7f09b5dff700 1 -- 172.21.15.7:0/4019110236 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09b0101960 msgr2=0x7f09b0101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:08.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.587+0000 7f09b5dff700 1 --2- 172.21.15.7:0/4019110236 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09b0101960 0x7f09b0101dc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f09a40099c0 tx=0x7f09a4009cd0 comp rx=0 tx=0).stop 2023-07-20T02:51:08.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.587+0000 7f09b5dff700 1 -- 172.21.15.7:0/4019110236 shutdown_connections 2023-07-20T02:51:08.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.587+0000 7f09b5dff700 1 --2- 172.21.15.7:0/4019110236 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09b0102300 0x7f09b0108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:08.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.587+0000 7f09b5dff700 1 --2- 172.21.15.7:0/4019110236 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09b0100770 0x7f09b0100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:08.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.587+0000 7f09b5dff700 1 --2- 172.21.15.7:0/4019110236 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09b0101960 0x7f09b0101dc0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:08.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.587+0000 7f09b5dff700 1 -- 172.21.15.7:0/4019110236 >> 172.21.15.7:0/4019110236 conn(0x7f09b00fbca0 msgr2=0x7f09b00fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:08.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.587+0000 7f09b5dff700 1 -- 172.21.15.7:0/4019110236 shutdown_connections 2023-07-20T02:51:08.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.587+0000 7f09b5dff700 1 -- 172.21.15.7:0/4019110236 wait complete. 2023-07-20T02:51:08.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.589+0000 7f09b5dff700 1 Processor -- start 2023-07-20T02:51:08.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.589+0000 7f09b5dff700 1 -- start start 2023-07-20T02:51:08.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.589+0000 7f09b5dff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09b0100770 0x7f09b0196440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:08.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.590+0000 7f09b5dff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09b0101960 0x7f09b0196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:08.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.590+0000 7f09b5dff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09b0102300 0x7f09b019d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:08.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.590+0000 7f09b5dff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b019dfc0 con 0x7f09b0101960 2023-07-20T02:51:08.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.590+0000 7f09b5dff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b019e100 con 0x7f09b0102300 2023-07-20T02:51:08.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.590+0000 7f09b5dff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b019e290 con 0x7f09b0100770 2023-07-20T02:51:08.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.590+0000 7f09af7fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09b0100770 0x7f09b0196440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:08.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.590+0000 7f09af7fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09b0100770 0x7f09b0196440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.7:35456/0 (socket says 172.21.15.7:35456) 2023-07-20T02:51:08.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.590+0000 7f09af7fe700 1 -- 172.21.15.7:0/392933851 learned_addr learned my addr 172.21.15.7:0/392933851 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:08.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.590+0000 7f09aeffd700 1 --2- 172.21.15.7:0/392933851 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09b0101960 0x7f09b0196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:08.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.590+0000 7f09affff700 1 --2- 172.21.15.7:0/392933851 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09b0102300 0x7f09b019d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:08.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.590+0000 7f09af7fe700 1 -- 172.21.15.7:0/392933851 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09b0102300 msgr2=0x7f09b019d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:08.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.590+0000 7f09af7fe700 1 --2- 172.21.15.7:0/392933851 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09b0102300 0x7f09b019d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:08.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.590+0000 7f09af7fe700 1 -- 172.21.15.7:0/392933851 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09b0101960 msgr2=0x7f09b0196980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:08.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.590+0000 7f09af7fe700 1 --2- 172.21.15.7:0/392933851 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09b0101960 0x7f09b0196980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:08.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.590+0000 7f09af7fe700 1 -- 172.21.15.7:0/392933851 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a40095f0 con 0x7f09b0100770 2023-07-20T02:51:08.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.590+0000 7f09af7fe700 1 --2- 172.21.15.7:0/392933851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09b0100770 0x7f09b0196440 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f09a000e970 tx=0x7f09a000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:08.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.591+0000 7f09acff9700 1 -- 172.21.15.7:0/392933851 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a0009790 con 0x7f09b0100770 2023-07-20T02:51:08.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.591+0000 7f09acff9700 1 -- 172.21.15.7:0/392933851 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f09a000eec0 con 0x7f09b0100770 2023-07-20T02:51:08.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.591+0000 7f09b5dff700 1 -- 172.21.15.7:0/392933851 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09b019e570 con 0x7f09b0100770 2023-07-20T02:51:08.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.591+0000 7f09acff9700 1 -- 172.21.15.7:0/392933851 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a0003cf0 con 0x7f09b0100770 2023-07-20T02:51:08.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.591+0000 7f09b5dff700 1 -- 172.21.15.7:0/392933851 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b019ead0 con 0x7f09b0100770 2023-07-20T02:51:08.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.592+0000 7f09b5dff700 1 -- 172.21.15.7:0/392933851 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09b010bfe0 con 0x7f09b0100770 2023-07-20T02:51:08.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.594+0000 7f09acff9700 1 -- 172.21.15.7:0/392933851 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f09a000ccd0 con 0x7f09b0100770 2023-07-20T02:51:08.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.595+0000 7f09acff9700 1 --2- 172.21.15.7:0/392933851 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f09980a1220 0x7f09980a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:08.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.595+0000 7f09acff9700 1 -- 172.21.15.7:0/392933851 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f09a0014070 con 0x7f09b0100770 2023-07-20T02:51:08.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.595+0000 7f09aeffd700 1 --2- 172.21.15.7:0/392933851 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f09980a1220 0x7f09980a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:08.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.596+0000 7f09aeffd700 1 --2- 172.21.15.7:0/392933851 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f09980a1220 0x7f09980a36e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f09a400b580 tx=0x7f09a4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:08.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.599+0000 7f09acff9700 1 -- 172.21.15.7:0/392933851 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f09a0096210 con 0x7f09b0100770 2023-07-20T02:51:08.837 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:08 smithi188 ceph-mon[129553]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:08.838 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:08 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:08.838 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:08 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:08.838 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:08 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:08.838 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:08 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:51:08.838 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:08 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:51:08.838 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:08 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:51:08.838 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:08 smithi188 ceph-mon[129553]: Deploying daemon mds.c on smithi188 2023-07-20T02:51:08.839 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:08 smithi188 ceph-mon[129553]: mds.? [v2:172.21.15.188:6834/857155289,v1:172.21.15.188:6835/857155289] up:boot 2023-07-20T02:51:08.839 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:08 smithi188 ceph-mon[129553]: fsmap 6 up:standby 2023-07-20T02:51:08.839 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:08 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-20T02:51:08.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.851+0000 7f09b5dff700 1 -- 172.21.15.7:0/392933851 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09b010c200 con 0x7f09b0100770 2023-07-20T02:51:08.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.852+0000 7f09acff9700 1 -- 172.21.15.7:0/392933851 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5039 (secure 0 0 0) 0x7f09a0099830 con 0x7f09b0100770 2023-07-20T02:51:08.865 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:51:08.865 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/190105946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":190105946},{"type":"v1","addr":"172.21.15.7:6837","nonce":190105946}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/1435841351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":1435841351},{"type":"v1","addr":"172.21.15.188:6833","nonce":1435841351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/401861265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":401861265},{"type":"v1","addr":"172.21.15.158:6833","nonce":401861265}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/4064058311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":4064058311},{"type":"v1","addr":"172.21.15.7:6835","nonce":4064058311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3931487788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3931487788},{"type":"v1","addr":"172.21.15.158:6835","nonce":3931487788}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/857155289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":857155289},{"type":"v1","addr":"172.21.15.188:6835","nonce":857155289}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[]} 2023-07-20T02:51:08.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.854+0000 7f09b5dff700 1 -- 172.21.15.7:0/392933851 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f09980a1220 msgr2=0x7f09980a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:08.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.854+0000 7f09b5dff700 1 --2- 172.21.15.7:0/392933851 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f09980a1220 0x7f09980a36e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f09a400b580 tx=0x7f09a4005fb0 comp rx=0 tx=0).stop 2023-07-20T02:51:08.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.854+0000 7f09b5dff700 1 -- 172.21.15.7:0/392933851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09b0100770 msgr2=0x7f09b0196440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:08.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.854+0000 7f09b5dff700 1 --2- 172.21.15.7:0/392933851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09b0100770 0x7f09b0196440 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f09a000e970 tx=0x7f09a000ec80 comp rx=0 tx=0).stop 2023-07-20T02:51:08.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.854+0000 7f09b5dff700 1 -- 172.21.15.7:0/392933851 shutdown_connections 2023-07-20T02:51:08.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.854+0000 7f09b5dff700 1 --2- 172.21.15.7:0/392933851 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f09b0102300 0x7f09b019d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:08.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.854+0000 7f09b5dff700 1 --2- 172.21.15.7:0/392933851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09b0100770 0x7f09b0196440 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:08.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.854+0000 7f09b5dff700 1 --2- 172.21.15.7:0/392933851 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f09980a1220 0x7f09980a36e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:08.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.854+0000 7f09b5dff700 1 --2- 172.21.15.7:0/392933851 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09b0101960 0x7f09b0196980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:08.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.854+0000 7f09b5dff700 1 -- 172.21.15.7:0/392933851 >> 172.21.15.7:0/392933851 conn(0x7f09b00fbca0 msgr2=0x7f09b00fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:08.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.854+0000 7f09b5dff700 1 -- 172.21.15.7:0/392933851 shutdown_connections 2023-07-20T02:51:08.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:08.854+0000 7f09b5dff700 1 -- 172.21.15.7:0/392933851 wait complete. 2023-07-20T02:51:08.869 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 7 2023-07-20T02:51:08.884 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:08 smithi007 ceph-mon[126353]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:08.884 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:08 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:08.885 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:08 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:08.885 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:08 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:08.885 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:08 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:51:08.885 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:08 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:51:08.885 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:08 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:51:08.885 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:08 smithi007 ceph-mon[126353]: Deploying daemon mds.c on smithi188 2023-07-20T02:51:08.885 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:08 smithi007 ceph-mon[126353]: mds.? [v2:172.21.15.188:6834/857155289,v1:172.21.15.188:6835/857155289] up:boot 2023-07-20T02:51:08.886 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:08 smithi007 ceph-mon[126353]: fsmap 6 up:standby 2023-07-20T02:51:08.886 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:08 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-20T02:51:08.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:08 smithi158 ceph-mon[128925]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:08.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:08 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:08.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:08 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:08.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:08 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:08.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:08 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:51:08.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:08 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:51:08.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:08 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:51:08.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:08 smithi158 ceph-mon[128925]: Deploying daemon mds.c on smithi188 2023-07-20T02:51:08.892 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:08 smithi158 ceph-mon[128925]: mds.? [v2:172.21.15.188:6834/857155289,v1:172.21.15.188:6835/857155289] up:boot 2023-07-20T02:51:08.892 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:08 smithi158 ceph-mon[128925]: fsmap 6 up:standby 2023-07-20T02:51:08.892 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:08 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-20T02:51:09.457 INFO:journalctl@ceph.mds.c.smithi188.stdout:Jul 20 02:51:09 smithi188 systemd[1]: Starting Ceph mds.c for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:51:09.716 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:09 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/392933851' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:09.716 INFO:journalctl@ceph.mds.c.smithi188.stdout:Jul 20 02:51:09 smithi188 podman[155624]: 2023-07-20T02:51:09.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:09 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/392933851' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:09.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:09 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/392933851' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:10.213 INFO:journalctl@ceph.mds.c.smithi188.stdout:Jul 20 02:51:09 smithi188 ceph-mds[155664]: -- 172.21.15.188:0/336411115 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f4193551e0 con 0x55f41932ac00 2023-07-20T02:51:10.214 INFO:journalctl@ceph.mds.c.smithi188.stdout:Jul 20 02:51:09 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-c[155640]: starting mds.c at 2023-07-20T02:51:10.214 INFO:journalctl@ceph.mds.c.smithi188.stdout:Jul 20 02:51:09 smithi188 bash[155624]: 2667a58ba5c5ce0de2d84f63f5d6df5cc6cb8eec6a07129fd902a4e4cd787a92 2023-07-20T02:51:10.214 INFO:journalctl@ceph.mds.c.smithi188.stdout:Jul 20 02:51:09 smithi188 systemd[1]: Started Ceph mds.c for 1b04643c-26a7-11ee-9b34-001a4aab830c. 2023-07-20T02:51:10.786 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:10 smithi158 ceph-mon[128925]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:10.787 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:10 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:10.787 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:10 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:10.788 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:10 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:10.788 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:10 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:51:10.788 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:10 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:51:10.789 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:10 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:51:10.888 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T02:51:10.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:10 smithi007 ceph-mon[126353]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:10.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:10 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:10.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:10 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:10.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:10 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:10.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:10 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:51:10.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:10 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:51:10.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:10 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:51:10.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:10 smithi188 ceph-mon[129553]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:10.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:10 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:10.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:10 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:10.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:10 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:10.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:10 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:51:10.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:10 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:51:10.965 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:10 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:51:11.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.090+0000 7f3f037ff700 1 -- 172.21.15.7:0/436944322 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3efc102310 msgr2=0x7f3efc108bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:11.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.090+0000 7f3f037ff700 1 --2- 172.21.15.7:0/436944322 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3efc102310 0x7f3efc108bc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3ef4009fc0 tx=0x7f3ef400b330 comp rx=0 tx=0).stop 2023-07-20T02:51:11.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.090+0000 7f3f037ff700 1 -- 172.21.15.7:0/436944322 shutdown_connections 2023-07-20T02:51:11.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.090+0000 7f3f037ff700 1 --2- 172.21.15.7:0/436944322 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3efc102310 0x7f3efc108bc0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:11.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.090+0000 7f3f037ff700 1 --2- 172.21.15.7:0/436944322 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3efc100780 0x7f3efc100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:11.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.090+0000 7f3f037ff700 1 --2- 172.21.15.7:0/436944322 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3efc101970 0x7f3efc101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:11.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.090+0000 7f3f037ff700 1 -- 172.21.15.7:0/436944322 >> 172.21.15.7:0/436944322 conn(0x7f3efc0fbcb0 msgr2=0x7f3efc0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:11.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.090+0000 7f3f037ff700 1 -- 172.21.15.7:0/436944322 shutdown_connections 2023-07-20T02:51:11.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.090+0000 7f3f037ff700 1 -- 172.21.15.7:0/436944322 wait complete. 2023-07-20T02:51:11.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.091+0000 7f3f037ff700 1 Processor -- start 2023-07-20T02:51:11.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.091+0000 7f3f037ff700 1 -- start start 2023-07-20T02:51:11.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.092+0000 7f3f037ff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3efc100780 0x7f3efc198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:11.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.092+0000 7f3f037ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3efc101970 0x7f3efc198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:11.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.092+0000 7f3f037ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3efc102310 0x7f3efc19fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:11.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.092+0000 7f3f037ff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3efc1a02c0 con 0x7f3efc102310 2023-07-20T02:51:11.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.092+0000 7f3f037ff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3efc1a0400 con 0x7f3efc100780 2023-07-20T02:51:11.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.092+0000 7f3f037ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3efc1a0590 con 0x7f3efc101970 2023-07-20T02:51:11.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.092+0000 7f3f0159b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3efc100780 0x7f3efc198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:11.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.092+0000 7f3f00d9a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3efc101970 0x7f3efc198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:11.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.092+0000 7f3f01d9c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3efc102310 0x7f3efc19fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:11.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.092+0000 7f3f0159b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3efc100780 0x7f3efc198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.7:34674/0 (socket says 172.21.15.7:34674) 2023-07-20T02:51:11.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.092+0000 7f3f0159b700 1 -- 172.21.15.7:0/1444114919 learned_addr learned my addr 172.21.15.7:0/1444114919 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:11.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.092+0000 7f3f01d9c700 1 -- 172.21.15.7:0/1444114919 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3efc101970 msgr2=0x7f3efc198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:11.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.092+0000 7f3f01d9c700 1 --2- 172.21.15.7:0/1444114919 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3efc101970 0x7f3efc198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:11.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.092+0000 7f3f01d9c700 1 -- 172.21.15.7:0/1444114919 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3efc100780 msgr2=0x7f3efc198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:11.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.092+0000 7f3f01d9c700 1 --2- 172.21.15.7:0/1444114919 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3efc100780 0x7f3efc198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:11.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.092+0000 7f3f01d9c700 1 -- 172.21.15.7:0/1444114919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ef80095f0 con 0x7f3efc102310 2023-07-20T02:51:11.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.092+0000 7f3f0159b700 1 --2- 172.21.15.7:0/1444114919 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3efc100780 0x7f3efc198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T02:51:11.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.092+0000 7f3f00d9a700 1 --2- 172.21.15.7:0/1444114919 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3efc101970 0x7f3efc198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T02:51:11.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.093+0000 7f3f01d9c700 1 --2- 172.21.15.7:0/1444114919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3efc102310 0x7f3efc19fcf0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3ef400b6c0 tx=0x7f3ef40077a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:11.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.093+0000 7f3ef27fc700 1 -- 172.21.15.7:0/1444114919 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef400e040 con 0x7f3efc102310 2023-07-20T02:51:11.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.093+0000 7f3ef27fc700 1 -- 172.21.15.7:0/1444114919 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3ef401d470 con 0x7f3efc102310 2023-07-20T02:51:11.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.093+0000 7f3ef27fc700 1 -- 172.21.15.7:0/1444114919 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef40052a0 con 0x7f3efc102310 2023-07-20T02:51:11.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.093+0000 7f3f037ff700 1 -- 172.21.15.7:0/1444114919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ef4009c40 con 0x7f3efc102310 2023-07-20T02:51:11.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.093+0000 7f3f037ff700 1 -- 172.21.15.7:0/1444114919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3efc1a0b00 con 0x7f3efc102310 2023-07-20T02:51:11.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.094+0000 7f3f037ff700 1 -- 172.21.15.7:0/1444114919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3efc10bff0 con 0x7f3efc102310 2023-07-20T02:51:11.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.096+0000 7f3ef27fc700 1 -- 172.21.15.7:0/1444114919 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f3ef401d5e0 con 0x7f3efc102310 2023-07-20T02:51:11.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.097+0000 7f3ef27fc700 1 --2- 172.21.15.7:0/1444114919 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f3ee80a1220 0x7f3ee80a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:11.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.098+0000 7f3ef27fc700 1 -- 172.21.15.7:0/1444114919 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3ef40cd850 con 0x7f3efc102310 2023-07-20T02:51:11.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.098+0000 7f3f0159b700 1 --2- 172.21.15.7:0/1444114919 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f3ee80a1220 0x7f3ee80a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:11.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.098+0000 7f3f0159b700 1 --2- 172.21.15.7:0/1444114919 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f3ee80a1220 0x7f3ee80a36e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3ef8005950 tx=0x7f3ef8009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:11.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.102+0000 7f3ef27fc700 1 -- 172.21.15.7:0/1444114919 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3ef4097430 con 0x7f3efc102310 2023-07-20T02:51:11.314 INFO:journalctl@ceph.mds.h.smithi158.stdout:Jul 20 02:51:11 smithi158 systemd[1]: Starting Ceph mds.h for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:51:11.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.355+0000 7f3f037ff700 1 -- 172.21.15.7:0/1444114919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3efc10c1b0 con 0x7f3efc102310 2023-07-20T02:51:11.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.356+0000 7f3ef27fc700 1 -- 172.21.15.7:0/1444114919 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+5796 (secure 0 0 0) 0x7f3ef409aa50 con 0x7f3efc102310 2023-07-20T02:51:11.369 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:51:11.369 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/190105946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":190105946},{"type":"v1","addr":"172.21.15.7:6837","nonce":190105946}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/1435841351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":1435841351},{"type":"v1","addr":"172.21.15.188:6833","nonce":1435841351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/401861265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":401861265},{"type":"v1","addr":"172.21.15.158:6833","nonce":401861265}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/4064058311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":4064058311},{"type":"v1","addr":"172.21.15.7:6835","nonce":4064058311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3931487788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3931487788},{"type":"v1","addr":"172.21.15.158:6835","nonce":3931487788}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/857155289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":857155289},{"type":"v1","addr":"172.21.15.188:6835","nonce":857155289}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/775436728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":775436728},{"type":"v1","addr":"172.21.15.188:6837","nonce":775436728}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2023-07-20T02:51:11.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.358+0000 7f3f037ff700 1 -- 172.21.15.7:0/1444114919 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f3ee80a1220 msgr2=0x7f3ee80a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:11.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.358+0000 7f3f037ff700 1 --2- 172.21.15.7:0/1444114919 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f3ee80a1220 0x7f3ee80a36e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3ef8005950 tx=0x7f3ef8009310 comp rx=0 tx=0).stop 2023-07-20T02:51:11.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.358+0000 7f3f037ff700 1 -- 172.21.15.7:0/1444114919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3efc102310 msgr2=0x7f3efc19fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:11.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.358+0000 7f3f037ff700 1 --2- 172.21.15.7:0/1444114919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3efc102310 0x7f3efc19fcf0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3ef400b6c0 tx=0x7f3ef40077a0 comp rx=0 tx=0).stop 2023-07-20T02:51:11.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.358+0000 7f3f037ff700 1 -- 172.21.15.7:0/1444114919 shutdown_connections 2023-07-20T02:51:11.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.358+0000 7f3f037ff700 1 --2- 172.21.15.7:0/1444114919 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f3ee80a1220 0x7f3ee80a36e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:11.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.358+0000 7f3f037ff700 1 --2- 172.21.15.7:0/1444114919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3efc102310 0x7f3efc19fcf0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:11.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.358+0000 7f3f037ff700 1 --2- 172.21.15.7:0/1444114919 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3efc100780 0x7f3efc198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:11.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.358+0000 7f3f037ff700 1 --2- 172.21.15.7:0/1444114919 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3efc101970 0x7f3efc198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:11.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.358+0000 7f3f037ff700 1 -- 172.21.15.7:0/1444114919 >> 172.21.15.7:0/1444114919 conn(0x7f3efc0fbcb0 msgr2=0x7f3efc0fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:11.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.359+0000 7f3f037ff700 1 -- 172.21.15.7:0/1444114919 shutdown_connections 2023-07-20T02:51:11.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:11.359+0000 7f3f037ff700 1 -- 172.21.15.7:0/1444114919 wait complete. 2023-07-20T02:51:11.373 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 8 2023-07-20T02:51:11.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:11 smithi007 ceph-mon[126353]: Deploying daemon mds.h on smithi158 2023-07-20T02:51:11.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:11 smithi007 ceph-mon[126353]: mds.? [v2:172.21.15.188:6836/775436728,v1:172.21.15.188:6837/775436728] up:boot 2023-07-20T02:51:11.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:11 smithi007 ceph-mon[126353]: fsmap 7 up:standby 2023-07-20T02:51:11.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:11 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-20T02:51:11.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:11 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/1444114919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:11.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:11 smithi158 ceph-mon[128925]: Deploying daemon mds.h on smithi158 2023-07-20T02:51:11.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:11 smithi158 ceph-mon[128925]: mds.? [v2:172.21.15.188:6836/775436728,v1:172.21.15.188:6837/775436728] up:boot 2023-07-20T02:51:11.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:11 smithi158 ceph-mon[128925]: fsmap 7 up:standby 2023-07-20T02:51:11.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:11 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-20T02:51:11.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:11 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/1444114919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:11.891 INFO:journalctl@ceph.mds.h.smithi158.stdout:Jul 20 02:51:11 smithi158 podman[153859]: 2023-07-20T02:51:11.892 INFO:journalctl@ceph.mds.h.smithi158.stdout:Jul 20 02:51:11 smithi158 ceph-mds[153899]: -- 172.21.15.158:0/781740395 <== mon.2 v2:172.21.15.158:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fdf62611e0 con 0x55fdf6236800 2023-07-20T02:51:11.892 INFO:journalctl@ceph.mds.h.smithi158.stdout:Jul 20 02:51:11 smithi158 bash[153859]: 70a7537d7d8c96622a274e2b7ed216423a2bef0e6de6d0975e894feb748d8d1d 2023-07-20T02:51:11.892 INFO:journalctl@ceph.mds.h.smithi158.stdout:Jul 20 02:51:11 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-h[153875]: starting mds.h at 2023-07-20T02:51:11.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:11 smithi188 ceph-mon[129553]: Deploying daemon mds.h on smithi158 2023-07-20T02:51:11.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:11 smithi188 ceph-mon[129553]: mds.? [v2:172.21.15.188:6836/775436728,v1:172.21.15.188:6837/775436728] up:boot 2023-07-20T02:51:11.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:11 smithi188 ceph-mon[129553]: fsmap 7 up:standby 2023-07-20T02:51:11.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:11 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-20T02:51:11.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:11 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/1444114919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:12.390 INFO:journalctl@ceph.mds.h.smithi158.stdout:Jul 20 02:51:11 smithi158 systemd[1]: Started Ceph mds.h for 1b04643c-26a7-11ee-9b34-001a4aab830c. 2023-07-20T02:51:12.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:12 smithi158 ceph-mon[128925]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:12.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:12 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:12.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:12 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:12.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:12 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:12.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:12 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:51:12.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:12 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:51:12.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:12 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:51:12.927 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:12 smithi188 ceph-mon[129553]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:12.927 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:12 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:12.927 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:12 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:12.927 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:12 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:12.927 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:12 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:51:12.927 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:12 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:51:12.928 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:12 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:51:12.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:12 smithi007 ceph-mon[126353]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:12.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:12 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:12.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:12 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:12.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:12 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:12.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:12 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:51:12.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:12 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:51:12.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:12 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:51:13.391 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T02:51:13.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.594+0000 7f868edff700 1 -- 172.21.15.7:0/849268004 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8688101960 msgr2=0x7f8688101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:13.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.594+0000 7f868edff700 1 --2- 172.21.15.7:0/849268004 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8688101960 0x7f8688101dc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f86740098f0 tx=0x7f8674009c00 comp rx=0 tx=0).stop 2023-07-20T02:51:13.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.595+0000 7f868edff700 1 -- 172.21.15.7:0/849268004 shutdown_connections 2023-07-20T02:51:13.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.595+0000 7f868edff700 1 --2- 172.21.15.7:0/849268004 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8688102300 0x7f8688108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:13.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.595+0000 7f868edff700 1 --2- 172.21.15.7:0/849268004 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8688100770 0x7f8688100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:13.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.595+0000 7f868edff700 1 --2- 172.21.15.7:0/849268004 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8688101960 0x7f8688101dc0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:13.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.595+0000 7f868edff700 1 -- 172.21.15.7:0/849268004 >> 172.21.15.7:0/849268004 conn(0x7f86880fbca0 msgr2=0x7f86880fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:13.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.595+0000 7f868edff700 1 -- 172.21.15.7:0/849268004 shutdown_connections 2023-07-20T02:51:13.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.595+0000 7f868edff700 1 -- 172.21.15.7:0/849268004 wait complete. 2023-07-20T02:51:13.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.596+0000 7f868edff700 1 Processor -- start 2023-07-20T02:51:13.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.596+0000 7f868edff700 1 -- start start 2023-07-20T02:51:13.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.596+0000 7f868edff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8688100770 0x7f86881964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:13.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.596+0000 7f868edff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8688101960 0x7f86881969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:13.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.596+0000 7f868edff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8688102300 0x7f868819da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:13.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.596+0000 7f868edff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868819e030 con 0x7f8688102300 2023-07-20T02:51:13.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.596+0000 7f868edff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868819e170 con 0x7f8688101960 2023-07-20T02:51:13.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.596+0000 7f868edff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868819e300 con 0x7f8688100770 2023-07-20T02:51:13.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.596+0000 7f868cb9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8688100770 0x7f86881964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:13.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.596+0000 7f868d39c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8688102300 0x7f868819da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:13.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.596+0000 7f868cb9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8688100770 0x7f86881964b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.7:48736/0 (socket says 172.21.15.7:48736) 2023-07-20T02:51:13.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.596+0000 7f868d39c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8688102300 0x7f868819da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43734/0 (socket says 172.21.15.7:43734) 2023-07-20T02:51:13.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.596+0000 7f868cb9b700 1 -- 172.21.15.7:0/3276065336 learned_addr learned my addr 172.21.15.7:0/3276065336 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:13.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.596+0000 7f8687fff700 1 --2- 172.21.15.7:0/3276065336 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8688101960 0x7f86881969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:13.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.597+0000 7f868cb9b700 1 -- 172.21.15.7:0/3276065336 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8688101960 msgr2=0x7f86881969f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:13.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.597+0000 7f868cb9b700 1 --2- 172.21.15.7:0/3276065336 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8688101960 0x7f86881969f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:13.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.597+0000 7f868cb9b700 1 -- 172.21.15.7:0/3276065336 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8688102300 msgr2=0x7f868819da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:13.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.597+0000 7f868cb9b700 1 --2- 172.21.15.7:0/3276065336 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8688102300 0x7f868819da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:13.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.597+0000 7f868cb9b700 1 -- 172.21.15.7:0/3276065336 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8674009520 con 0x7f8688100770 2023-07-20T02:51:13.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.597+0000 7f868cb9b700 1 --2- 172.21.15.7:0/3276065336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8688100770 0x7f86881964b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f867c00e9f0 tx=0x7f867c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:13.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.597+0000 7f8685ffb700 1 -- 172.21.15.7:0/3276065336 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c004220 con 0x7f8688100770 2023-07-20T02:51:13.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.598+0000 7f868edff700 1 -- 172.21.15.7:0/3276065336 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8688072a00 con 0x7f8688100770 2023-07-20T02:51:13.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.598+0000 7f8685ffb700 1 -- 172.21.15.7:0/3276065336 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f867c004380 con 0x7f8688100770 2023-07-20T02:51:13.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.598+0000 7f8685ffb700 1 -- 172.21.15.7:0/3276065336 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c003b10 con 0x7f8688100770 2023-07-20T02:51:13.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.598+0000 7f868edff700 1 -- 172.21.15.7:0/3276065336 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8688072f60 con 0x7f8688100770 2023-07-20T02:51:13.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.599+0000 7f868edff700 1 -- 172.21.15.7:0/3276065336 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8688107d60 con 0x7f8688100770 2023-07-20T02:51:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.601+0000 7f8685ffb700 1 -- 172.21.15.7:0/3276065336 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f867c003c70 con 0x7f8688100770 2023-07-20T02:51:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.602+0000 7f8685ffb700 1 --2- 172.21.15.7:0/3276065336 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f86780a11a0 0x7f86780a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.602+0000 7f8685ffb700 1 -- 172.21.15.7:0/3276065336 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f867c014070 con 0x7f8688100770 2023-07-20T02:51:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.602+0000 7f8687fff700 1 --2- 172.21.15.7:0/3276065336 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f86780a11a0 0x7f86780a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.603+0000 7f8687fff700 1 --2- 172.21.15.7:0/3276065336 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f86780a11a0 0x7f86780a3660 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8674005e90 tx=0x7f8674019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:13.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.606+0000 7f8685ffb700 1 -- 172.21.15.7:0/3276065336 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f867c096150 con 0x7f8688100770 2023-07-20T02:51:13.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.862+0000 7f868edff700 1 -- 172.21.15.7:0/3276065336 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f868802cc00 con 0x7f8688100770 2023-07-20T02:51:13.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.863+0000 7f8685ffb700 1 -- 172.21.15.7:0/3276065336 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6556 (secure 0 0 0) 0x7f867c099770 con 0x7f8688100770 2023-07-20T02:51:13.877 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:51:13.877 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/190105946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":190105946},{"type":"v1","addr":"172.21.15.7:6837","nonce":190105946}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/1435841351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":1435841351},{"type":"v1","addr":"172.21.15.188:6833","nonce":1435841351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/401861265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":401861265},{"type":"v1","addr":"172.21.15.158:6833","nonce":401861265}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/4064058311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":4064058311},{"type":"v1","addr":"172.21.15.7:6835","nonce":4064058311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3931487788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3931487788},{"type":"v1","addr":"172.21.15.158:6835","nonce":3931487788}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/857155289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":857155289},{"type":"v1","addr":"172.21.15.188:6835","nonce":857155289}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1032497949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6836","nonce":1032497949},{"type":"v1","addr":"172.21.15.158:6837","nonce":1032497949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/775436728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":775436728},{"type":"v1","addr":"172.21.15.188:6837","nonce":775436728}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2023-07-20T02:51:13.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.866+0000 7f868edff700 1 -- 172.21.15.7:0/3276065336 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f86780a11a0 msgr2=0x7f86780a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:13.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.866+0000 7f868edff700 1 --2- 172.21.15.7:0/3276065336 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f86780a11a0 0x7f86780a3660 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8674005e90 tx=0x7f8674019040 comp rx=0 tx=0).stop 2023-07-20T02:51:13.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.866+0000 7f868edff700 1 -- 172.21.15.7:0/3276065336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8688100770 msgr2=0x7f86881964b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:13.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.866+0000 7f868edff700 1 --2- 172.21.15.7:0/3276065336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8688100770 0x7f86881964b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f867c00e9f0 tx=0x7f867c00ed00 comp rx=0 tx=0).stop 2023-07-20T02:51:13.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.866+0000 7f868edff700 1 -- 172.21.15.7:0/3276065336 shutdown_connections 2023-07-20T02:51:13.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.866+0000 7f868edff700 1 --2- 172.21.15.7:0/3276065336 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f86780a11a0 0x7f86780a3660 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:13.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.866+0000 7f868edff700 1 --2- 172.21.15.7:0/3276065336 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8688102300 0x7f868819da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:13.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.866+0000 7f868edff700 1 --2- 172.21.15.7:0/3276065336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8688100770 0x7f86881964b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:13.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.866+0000 7f868edff700 1 --2- 172.21.15.7:0/3276065336 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f8688101960 0x7f86881969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:13.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.866+0000 7f868edff700 1 -- 172.21.15.7:0/3276065336 >> 172.21.15.7:0/3276065336 conn(0x7f86880fbca0 msgr2=0x7f8688104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:13.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.866+0000 7f868edff700 1 -- 172.21.15.7:0/3276065336 shutdown_connections 2023-07-20T02:51:13.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:13.866+0000 7f868edff700 1 -- 172.21.15.7:0/3276065336 wait complete. 2023-07-20T02:51:13.881 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 9 2023-07-20T02:51:13.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:13 smithi158 ceph-mon[128925]: Deploying daemon mds.i on smithi188 2023-07-20T02:51:13.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:13 smithi158 ceph-mon[128925]: mds.? [v2:172.21.15.158:6836/1032497949,v1:172.21.15.158:6837/1032497949] up:boot 2023-07-20T02:51:13.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:13 smithi158 ceph-mon[128925]: fsmap 8 up:standby 2023-07-20T02:51:13.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:13 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-20T02:51:13.895 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:13 smithi007 ceph-mon[126353]: Deploying daemon mds.i on smithi188 2023-07-20T02:51:13.895 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:13 smithi007 ceph-mon[126353]: mds.? [v2:172.21.15.158:6836/1032497949,v1:172.21.15.158:6837/1032497949] up:boot 2023-07-20T02:51:13.896 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:13 smithi007 ceph-mon[126353]: fsmap 8 up:standby 2023-07-20T02:51:13.896 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:13 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-20T02:51:13.898 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:13 smithi188 ceph-mon[129553]: Deploying daemon mds.i on smithi188 2023-07-20T02:51:13.898 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:13 smithi188 ceph-mon[129553]: mds.? [v2:172.21.15.158:6836/1032497949,v1:172.21.15.158:6837/1032497949] up:boot 2023-07-20T02:51:13.898 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:13 smithi188 ceph-mon[129553]: fsmap 8 up:standby 2023-07-20T02:51:13.898 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:13 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-20T02:51:13.899 INFO:journalctl@ceph.mds.i.smithi188.stdout:Jul 20 02:51:13 smithi188 systemd[1]: Starting Ceph mds.i for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:51:14.179 INFO:journalctl@ceph.mds.i.smithi188.stdout:Jul 20 02:51:13 smithi188 podman[155965]: 2023-07-20T02:51:14.463 INFO:journalctl@ceph.mds.i.smithi188.stdout:Jul 20 02:51:14 smithi188 ceph-mds[156005]: -- 172.21.15.188:0/3524788909 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55aa609091e0 con 0x55aa608de800 2023-07-20T02:51:14.464 INFO:journalctl@ceph.mds.i.smithi188.stdout:Jul 20 02:51:14 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-i[155981]: starting mds.i at 2023-07-20T02:51:14.464 INFO:journalctl@ceph.mds.i.smithi188.stdout:Jul 20 02:51:14 smithi188 bash[155965]: 2427985e5b43e83a728892010e4f4a58a4c72fcbec4e17f474507290bd9fd95c 2023-07-20T02:51:14.464 INFO:journalctl@ceph.mds.i.smithi188.stdout:Jul 20 02:51:14 smithi188 systemd[1]: Started Ceph mds.i for 1b04643c-26a7-11ee-9b34-001a4aab830c. 2023-07-20T02:51:14.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:14 smithi158 ceph-mon[128925]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:14.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:14 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/3276065336' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:14.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:14 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:14.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:14 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:14.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:14 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:14.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:14 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:51:14.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:14 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:51:14.892 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:14 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:51:14.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:14 smithi007 ceph-mon[126353]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:14.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:14 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/3276065336' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:14.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:14 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:14.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:14 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:14.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:14 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:14.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:14 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:51:14.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:14 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:51:14.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:14 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:51:14.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:14 smithi188 ceph-mon[129553]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:14.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:14 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/3276065336' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:14.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:14 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:14.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:14 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:14.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:14 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:14.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:14 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:51:14.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:14 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:51:14.965 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:14 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:51:15.866 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:15 smithi007 ceph-mon[126353]: Deploying daemon mds.g on smithi007 2023-07-20T02:51:15.866 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:15 smithi007 ceph-mon[126353]: mds.? [v2:172.21.15.188:6838/450074084,v1:172.21.15.188:6839/450074084] up:boot 2023-07-20T02:51:15.866 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:15 smithi007 ceph-mon[126353]: fsmap 9 up:standby 2023-07-20T02:51:15.866 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:15 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-20T02:51:15.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:15 smithi158 ceph-mon[128925]: Deploying daemon mds.g on smithi007 2023-07-20T02:51:15.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:15 smithi158 ceph-mon[128925]: mds.? [v2:172.21.15.188:6838/450074084,v1:172.21.15.188:6839/450074084] up:boot 2023-07-20T02:51:15.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:15 smithi158 ceph-mon[128925]: fsmap 9 up:standby 2023-07-20T02:51:15.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:15 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-20T02:51:15.899 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T02:51:15.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:15 smithi188 ceph-mon[129553]: Deploying daemon mds.g on smithi007 2023-07-20T02:51:15.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:15 smithi188 ceph-mon[129553]: mds.? [v2:172.21.15.188:6838/450074084,v1:172.21.15.188:6839/450074084] up:boot 2023-07-20T02:51:15.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:15 smithi188 ceph-mon[129553]: fsmap 9 up:standby 2023-07-20T02:51:15.965 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:15 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-20T02:51:16.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.320+0000 7f9fe5fff700 1 -- 172.21.15.7:0/736138014 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9fe0101930 msgr2=0x7f9fe0103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:16.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.320+0000 7f9fe5fff700 1 --2- 172.21.15.7:0/736138014 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9fe0101930 0x7f9fe0103dc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9fd40098f0 tx=0x7f9fd4009c00 comp rx=0 tx=0).stop 2023-07-20T02:51:16.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.321+0000 7f9fe5fff700 1 -- 172.21.15.7:0/736138014 shutdown_connections 2023-07-20T02:51:16.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.321+0000 7f9fe5fff700 1 --2- 172.21.15.7:0/736138014 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9fe0104300 0x7f9fe0104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:16.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.321+0000 7f9fe5fff700 1 --2- 172.21.15.7:0/736138014 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9fe00fefc0 0x7f9fe01013f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:16.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.321+0000 7f9fe5fff700 1 --2- 172.21.15.7:0/736138014 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9fe0101930 0x7f9fe0103dc0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:16.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.321+0000 7f9fe5fff700 1 -- 172.21.15.7:0/736138014 >> 172.21.15.7:0/736138014 conn(0x7f9fe00faba0 msgr2=0x7f9fe00fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:16.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.321+0000 7f9fe5fff700 1 -- 172.21.15.7:0/736138014 shutdown_connections 2023-07-20T02:51:16.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.321+0000 7f9fe5fff700 1 -- 172.21.15.7:0/736138014 wait complete. 2023-07-20T02:51:16.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.322+0000 7f9fe5fff700 1 Processor -- start 2023-07-20T02:51:16.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.322+0000 7f9fe5fff700 1 -- start start 2023-07-20T02:51:16.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.322+0000 7f9fe5fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9fe00fefc0 0x7f9fe019cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:16.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.322+0000 7f9fe5fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9fe0101930 0x7f9fe019cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:16.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.322+0000 7f9fe5fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9fe0104300 0x7f9fe01a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:16.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.322+0000 7f9fe5fff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe01a4630 con 0x7f9fe0104300 2023-07-20T02:51:16.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.322+0000 7f9fe5fff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe01a4770 con 0x7f9fe0101930 2023-07-20T02:51:16.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.322+0000 7f9fe5fff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe01a4900 con 0x7f9fe00fefc0 2023-07-20T02:51:16.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.323+0000 7f9fdffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9fe0104300 0x7f9fe01a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:16.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.323+0000 7f9fdeffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9fe0101930 0x7f9fe019cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:16.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.323+0000 7f9fdf7fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9fe00fefc0 0x7f9fe019cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:16.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.323+0000 7f9fdeffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9fe0101930 0x7f9fe019cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.7:38194/0 (socket says 172.21.15.7:38194) 2023-07-20T02:51:16.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.323+0000 7f9fdeffd700 1 -- 172.21.15.7:0/1486979232 learned_addr learned my addr 172.21.15.7:0/1486979232 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:16.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.323+0000 7f9fdeffd700 1 -- 172.21.15.7:0/1486979232 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9fe00fefc0 msgr2=0x7f9fe019cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:16.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.323+0000 7f9fdeffd700 1 --2- 172.21.15.7:0/1486979232 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9fe00fefc0 0x7f9fe019cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:16.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.323+0000 7f9fdeffd700 1 -- 172.21.15.7:0/1486979232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9fe0104300 msgr2=0x7f9fe01a4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:16.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.323+0000 7f9fdeffd700 1 --2- 172.21.15.7:0/1486979232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9fe0104300 0x7f9fe01a4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:16.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.323+0000 7f9fdeffd700 1 -- 172.21.15.7:0/1486979232 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd4009520 con 0x7f9fe0101930 2023-07-20T02:51:16.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.323+0000 7f9fdffff700 1 --2- 172.21.15.7:0/1486979232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9fe0104300 0x7f9fe01a4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T02:51:16.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.323+0000 7f9fdeffd700 1 --2- 172.21.15.7:0/1486979232 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9fe0101930 0x7f9fe019cff0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9fd4009fd0 tx=0x7f9fd400b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:16.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.324+0000 7f9fdcff9700 1 -- 172.21.15.7:0/1486979232 <== mon.2 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd401c070 con 0x7f9fe0101930 2023-07-20T02:51:16.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.324+0000 7f9fe5fff700 1 -- 172.21.15.7:0/1486979232 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fe01a4b80 con 0x7f9fe0101930 2023-07-20T02:51:16.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.324+0000 7f9fdcff9700 1 -- 172.21.15.7:0/1486979232 <== mon.2 v2:172.21.15.158:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9fd400f7d0 con 0x7f9fe0101930 2023-07-20T02:51:16.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.324+0000 7f9fdcff9700 1 -- 172.21.15.7:0/1486979232 <== mon.2 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd4004ab0 con 0x7f9fe0101930 2023-07-20T02:51:16.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.324+0000 7f9fe5fff700 1 -- 172.21.15.7:0/1486979232 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fe01a5000 con 0x7f9fe0101930 2023-07-20T02:51:16.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.325+0000 7f9fe5fff700 1 -- 172.21.15.7:0/1486979232 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fe00fc720 con 0x7f9fe0101930 2023-07-20T02:51:16.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.327+0000 7f9fdcff9700 1 -- 172.21.15.7:0/1486979232 <== mon.2 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f9fd4020070 con 0x7f9fe0101930 2023-07-20T02:51:16.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.329+0000 7f9fdcff9700 1 --2- 172.21.15.7:0/1486979232 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f9fc80a1220 0x7f9fc80a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:16.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.330+0000 7f9fdcff9700 1 -- 172.21.15.7:0/1486979232 <== mon.2 v2:172.21.15.158:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f9fd40ccff0 con 0x7f9fe0101930 2023-07-20T02:51:16.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.330+0000 7f9fdf7fe700 1 --2- 172.21.15.7:0/1486979232 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f9fc80a1220 0x7f9fc80a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:16.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.330+0000 7f9fdf7fe700 1 --2- 172.21.15.7:0/1486979232 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f9fc80a1220 0x7f9fc80a36e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9fd0005fd0 tx=0x7f9fd0009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:16.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.331+0000 7f9fdcff9700 1 -- 172.21.15.7:0/1486979232 <== mon.2 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9fd4096c30 con 0x7f9fe0101930 2023-07-20T02:51:16.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.616+0000 7f9fe5fff700 1 -- 172.21.15.7:0/1486979232 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9fe002c970 con 0x7f9fe0101930 2023-07-20T02:51:16.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.617+0000 7f9fdcff9700 1 -- 172.21.15.7:0/1486979232 <== mon.2 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7315 (secure 0 0 0) 0x7f9fd409a250 con 0x7f9fe0101930 2023-07-20T02:51:16.631 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:51:16.631 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/190105946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":190105946},{"type":"v1","addr":"172.21.15.7:6837","nonce":190105946}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/1435841351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":1435841351},{"type":"v1","addr":"172.21.15.188:6833","nonce":1435841351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/401861265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":401861265},{"type":"v1","addr":"172.21.15.158:6833","nonce":401861265}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/4064058311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":4064058311},{"type":"v1","addr":"172.21.15.7:6835","nonce":4064058311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3931487788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3931487788},{"type":"v1","addr":"172.21.15.158:6835","nonce":3931487788}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/857155289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":857155289},{"type":"v1","addr":"172.21.15.188:6835","nonce":857155289}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1032497949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6836","nonce":1032497949},{"type":"v1","addr":"172.21.15.158:6837","nonce":1032497949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/775436728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":775436728},{"type":"v1","addr":"172.21.15.188:6837","nonce":775436728}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24562,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6839/450074084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":450074084},{"type":"v1","addr":"172.21.15.188:6839","nonce":450074084}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-20T02:51:16.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.619+0000 7f9fe5fff700 1 -- 172.21.15.7:0/1486979232 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f9fc80a1220 msgr2=0x7f9fc80a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:16.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.620+0000 7f9fe5fff700 1 --2- 172.21.15.7:0/1486979232 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f9fc80a1220 0x7f9fc80a36e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9fd0005fd0 tx=0x7f9fd0009310 comp rx=0 tx=0).stop 2023-07-20T02:51:16.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.620+0000 7f9fe5fff700 1 -- 172.21.15.7:0/1486979232 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9fe0101930 msgr2=0x7f9fe019cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:16.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.620+0000 7f9fe5fff700 1 --2- 172.21.15.7:0/1486979232 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9fe0101930 0x7f9fe019cff0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9fd4009fd0 tx=0x7f9fd400b790 comp rx=0 tx=0).stop 2023-07-20T02:51:16.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.620+0000 7f9fe5fff700 1 -- 172.21.15.7:0/1486979232 shutdown_connections 2023-07-20T02:51:16.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.620+0000 7f9fe5fff700 1 --2- 172.21.15.7:0/1486979232 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f9fc80a1220 0x7f9fc80a36e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:16.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.620+0000 7f9fe5fff700 1 --2- 172.21.15.7:0/1486979232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9fe0104300 0x7f9fe01a4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:16.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.620+0000 7f9fe5fff700 1 --2- 172.21.15.7:0/1486979232 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9fe00fefc0 0x7f9fe019cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:16.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.620+0000 7f9fe5fff700 1 --2- 172.21.15.7:0/1486979232 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f9fe0101930 0x7f9fe019cff0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:16.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.620+0000 7f9fe5fff700 1 -- 172.21.15.7:0/1486979232 >> 172.21.15.7:0/1486979232 conn(0x7f9fe00faba0 msgr2=0x7f9fe00ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:16.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.620+0000 7f9fe5fff700 1 -- 172.21.15.7:0/1486979232 shutdown_connections 2023-07-20T02:51:16.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:16.620+0000 7f9fe5fff700 1 -- 172.21.15.7:0/1486979232 wait complete. 2023-07-20T02:51:16.635 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 10 2023-07-20T02:51:16.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:16 smithi158 ceph-mon[128925]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:16.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:16 smithi007 ceph-mon[126353]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:16.949 INFO:journalctl@ceph.mds.g.smithi007.stdout:Jul 20 02:51:16 smithi007 systemd[1]: Starting Ceph mds.g for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:51:16.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:16 smithi188 ceph-mon[129553]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:17.312 INFO:journalctl@ceph.mds.g.smithi007.stdout:Jul 20 02:51:17 smithi007 podman[160981]: 2023-07-20T02:51:17.313 INFO:journalctl@ceph.mds.g.smithi007.stdout:Jul 20 02:51:17 smithi007 bash[160981]: 980e4b65ee2ce33e316c284a9f4980676922a72040c51ade29c1719b87fcb007 2023-07-20T02:51:17.313 INFO:journalctl@ceph.mds.g.smithi007.stdout:Jul 20 02:51:17 smithi007 ceph-mds[161021]: -- 172.21.15.7:0/1684474415 <== mon.0 v2:172.21.15.7:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558b337751e0 con 0x558b3374ac00 2023-07-20T02:51:17.313 INFO:journalctl@ceph.mds.g.smithi007.stdout:Jul 20 02:51:17 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-g[160997]: starting mds.g at 2023-07-20T02:51:17.620 INFO:journalctl@ceph.mds.g.smithi007.stdout:Jul 20 02:51:17 smithi007 systemd[1]: Started Ceph mds.g for 1b04643c-26a7-11ee-9b34-001a4aab830c. 2023-07-20T02:51:17.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:17 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/1486979232' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:17.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:17 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:17.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:17 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:17.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:17 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:17.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:17 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:51:17.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:17 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:51:17.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:17 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:51:17.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:17 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/1486979232' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:17.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:17 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:17.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:17 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:17.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:17 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:17.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:17 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:51:17.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:17 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:51:17.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:17 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:51:17.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:17 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/1486979232' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:17.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:17 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:17.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:17 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:17.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:17 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:17.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:17 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:51:17.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:17 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:51:17.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:17 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:51:18.655 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T02:51:18.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.852+0000 7f21947ff700 1 -- 172.21.15.7:0/664878272 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f218c1018f0 msgr2=0x7f218c101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:18.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.852+0000 7f21947ff700 1 --2- 172.21.15.7:0/664878272 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f218c1018f0 0x7f218c101d50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f217c0099c0 tx=0x7f217c009cd0 comp rx=0 tx=0).stop 2023-07-20T02:51:18.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.852+0000 7f21947ff700 1 -- 172.21.15.7:0/664878272 shutdown_connections 2023-07-20T02:51:18.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.852+0000 7f21947ff700 1 --2- 172.21.15.7:0/664878272 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f218c102290 0x7f218c108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:18.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.852+0000 7f21947ff700 1 --2- 172.21.15.7:0/664878272 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f218c100700 0x7f218c100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:18.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.852+0000 7f21947ff700 1 --2- 172.21.15.7:0/664878272 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f218c1018f0 0x7f218c101d50 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:18.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.852+0000 7f21947ff700 1 -- 172.21.15.7:0/664878272 >> 172.21.15.7:0/664878272 conn(0x7f218c0fbc50 msgr2=0x7f218c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:18.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.852+0000 7f21947ff700 1 -- 172.21.15.7:0/664878272 shutdown_connections 2023-07-20T02:51:18.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.852+0000 7f21947ff700 1 -- 172.21.15.7:0/664878272 wait complete. 2023-07-20T02:51:18.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.853+0000 7f21947ff700 1 Processor -- start 2023-07-20T02:51:18.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.853+0000 7f21947ff700 1 -- start start 2023-07-20T02:51:18.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.854+0000 7f21947ff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f218c100700 0x7f218c198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:18.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.854+0000 7f21947ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f218c1018f0 0x7f218c198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:18.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.854+0000 7f21947ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f218c102290 0x7f218c19fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:18.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.854+0000 7f21947ff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f218c1a0110 con 0x7f218c102290 2023-07-20T02:51:18.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.854+0000 7f21947ff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f218c1a0250 con 0x7f218c100700 2023-07-20T02:51:18.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.854+0000 7f21947ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f218c1a03e0 con 0x7f218c1018f0 2023-07-20T02:51:18.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.854+0000 7f2192d9c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f218c102290 0x7f218c19fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:18.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.854+0000 7f2192d9c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f218c102290 0x7f218c19fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43804/0 (socket says 172.21.15.7:43804) 2023-07-20T02:51:18.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.854+0000 7f2192d9c700 1 -- 172.21.15.7:0/289027775 learned_addr learned my addr 172.21.15.7:0/289027775 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:18.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.854+0000 7f219259b700 1 --2- 172.21.15.7:0/289027775 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f218c100700 0x7f218c198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:18.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.854+0000 7f2191d9a700 1 --2- 172.21.15.7:0/289027775 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f218c1018f0 0x7f218c198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:18.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.854+0000 7f2192d9c700 1 -- 172.21.15.7:0/289027775 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f218c1018f0 msgr2=0x7f218c198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:18.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.854+0000 7f2192d9c700 1 --2- 172.21.15.7:0/289027775 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f218c1018f0 0x7f218c198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:18.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.854+0000 7f2192d9c700 1 -- 172.21.15.7:0/289027775 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f218c100700 msgr2=0x7f218c198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:18.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.854+0000 7f2192d9c700 1 --2- 172.21.15.7:0/289027775 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f218c100700 0x7f218c198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:18.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.854+0000 7f2192d9c700 1 -- 172.21.15.7:0/289027775 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f217c0095f0 con 0x7f218c102290 2023-07-20T02:51:18.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.855+0000 7f2192d9c700 1 --2- 172.21.15.7:0/289027775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f218c102290 0x7f218c19fb40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2184007d70 tx=0x7f218400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:18.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.855+0000 7f21837fe700 1 -- 172.21.15.7:0/289027775 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f218400ac90 con 0x7f218c102290 2023-07-20T02:51:18.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.855+0000 7f21837fe700 1 -- 172.21.15.7:0/289027775 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f218400adf0 con 0x7f218c102290 2023-07-20T02:51:18.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.855+0000 7f21837fe700 1 -- 172.21.15.7:0/289027775 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21840115e0 con 0x7f218c102290 2023-07-20T02:51:18.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.855+0000 7f21947ff700 1 -- 172.21.15.7:0/289027775 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f218c1a06c0 con 0x7f218c102290 2023-07-20T02:51:18.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.855+0000 7f21947ff700 1 -- 172.21.15.7:0/289027775 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f218c1a0b80 con 0x7f218c102290 2023-07-20T02:51:18.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.856+0000 7f21947ff700 1 -- 172.21.15.7:0/289027775 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f218c10bf90 con 0x7f218c102290 2023-07-20T02:51:18.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.858+0000 7f21837fe700 1 -- 172.21.15.7:0/289027775 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f2184012420 con 0x7f218c102290 2023-07-20T02:51:18.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.859+0000 7f21837fe700 1 --2- 172.21.15.7:0/289027775 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f21780a1220 0x7f21780a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:18.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.860+0000 7f21837fe700 1 -- 172.21.15.7:0/289027775 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f21840cc680 con 0x7f218c102290 2023-07-20T02:51:18.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.860+0000 7f219259b700 1 --2- 172.21.15.7:0/289027775 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f21780a1220 0x7f21780a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:18.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.861+0000 7f219259b700 1 --2- 172.21.15.7:0/289027775 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f21780a1220 0x7f21780a36e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f218c101750 tx=0x7f2188009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:18.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:18.864+0000 7f21837fe700 1 -- 172.21.15.7:0/289027775 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f21840961d0 con 0x7f218c102290 2023-07-20T02:51:18.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:18 smithi158 ceph-mon[128925]: Deploying daemon mds.a on smithi007 2023-07-20T02:51:18.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:18 smithi158 ceph-mon[128925]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:18.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:18 smithi158 ceph-mon[128925]: mds.? [v2:172.21.15.7:6838/3310846795,v1:172.21.15.7:6839/3310846795] up:boot 2023-07-20T02:51:18.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:18 smithi158 ceph-mon[128925]: fsmap 10 up:standby 2023-07-20T02:51:18.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:18 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-20T02:51:18.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:18 smithi007 ceph-mon[126353]: Deploying daemon mds.a on smithi007 2023-07-20T02:51:18.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:18 smithi007 ceph-mon[126353]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:18.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:18 smithi007 ceph-mon[126353]: mds.? [v2:172.21.15.7:6838/3310846795,v1:172.21.15.7:6839/3310846795] up:boot 2023-07-20T02:51:18.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:18 smithi007 ceph-mon[126353]: fsmap 10 up:standby 2023-07-20T02:51:18.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:18 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-20T02:51:18.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:18 smithi188 ceph-mon[129553]: Deploying daemon mds.a on smithi007 2023-07-20T02:51:18.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:18 smithi188 ceph-mon[129553]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:18.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:18 smithi188 ceph-mon[129553]: mds.? [v2:172.21.15.7:6838/3310846795,v1:172.21.15.7:6839/3310846795] up:boot 2023-07-20T02:51:18.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:18 smithi188 ceph-mon[129553]: fsmap 10 up:standby 2023-07-20T02:51:18.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:18 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-20T02:51:19.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:19.130+0000 7f21947ff700 1 -- 172.21.15.7:0/289027775 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f218c10c180 con 0x7f218c102290 2023-07-20T02:51:19.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:19.131+0000 7f21837fe700 1 -- 172.21.15.7:0/289027775 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8070 (secure 0 0 0) 0x7f21840997f0 con 0x7f218c102290 2023-07-20T02:51:19.145 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:51:19.145 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/190105946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":190105946},{"type":"v1","addr":"172.21.15.7:6837","nonce":190105946}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/3310846795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":3310846795},{"type":"v1","addr":"172.21.15.7:6839","nonce":3310846795}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/1435841351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":1435841351},{"type":"v1","addr":"172.21.15.188:6833","nonce":1435841351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/401861265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":401861265},{"type":"v1","addr":"172.21.15.158:6833","nonce":401861265}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/4064058311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":4064058311},{"type":"v1","addr":"172.21.15.7:6835","nonce":4064058311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3931487788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3931487788},{"type":"v1","addr":"172.21.15.158:6835","nonce":3931487788}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/857155289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":857155289},{"type":"v1","addr":"172.21.15.188:6835","nonce":857155289}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1032497949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6836","nonce":1032497949},{"type":"v1","addr":"172.21.15.158:6837","nonce":1032497949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/775436728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":775436728},{"type":"v1","addr":"172.21.15.188:6837","nonce":775436728}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24562,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6839/450074084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":450074084},{"type":"v1","addr":"172.21.15.188:6839","nonce":450074084}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-20T02:51:19.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:19.133+0000 7f21947ff700 1 -- 172.21.15.7:0/289027775 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f21780a1220 msgr2=0x7f21780a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:19.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:19.133+0000 7f21947ff700 1 --2- 172.21.15.7:0/289027775 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f21780a1220 0x7f21780a36e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f218c101750 tx=0x7f2188009260 comp rx=0 tx=0).stop 2023-07-20T02:51:19.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:19.133+0000 7f21947ff700 1 -- 172.21.15.7:0/289027775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f218c102290 msgr2=0x7f218c19fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:19.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:19.133+0000 7f21947ff700 1 --2- 172.21.15.7:0/289027775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f218c102290 0x7f218c19fb40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2184007d70 tx=0x7f218400a340 comp rx=0 tx=0).stop 2023-07-20T02:51:19.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:19.133+0000 7f21947ff700 1 -- 172.21.15.7:0/289027775 shutdown_connections 2023-07-20T02:51:19.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:19.133+0000 7f21947ff700 1 --2- 172.21.15.7:0/289027775 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f21780a1220 0x7f21780a36e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:19.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:19.133+0000 7f21947ff700 1 --2- 172.21.15.7:0/289027775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f218c102290 0x7f218c19fb40 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:19.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:19.133+0000 7f21947ff700 1 --2- 172.21.15.7:0/289027775 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f218c100700 0x7f218c198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:19.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:19.133+0000 7f21947ff700 1 --2- 172.21.15.7:0/289027775 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f218c1018f0 0x7f218c198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:19.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:19.133+0000 7f21947ff700 1 -- 172.21.15.7:0/289027775 >> 172.21.15.7:0/289027775 conn(0x7f218c0fbc50 msgr2=0x7f218c107190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:19.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:19.134+0000 7f21947ff700 1 -- 172.21.15.7:0/289027775 shutdown_connections 2023-07-20T02:51:19.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:19.134+0000 7f21947ff700 1 -- 172.21.15.7:0/289027775 wait complete. 2023-07-20T02:51:19.149 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 11 2023-07-20T02:51:19.778 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:19 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/289027775' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:19.779 INFO:journalctl@ceph.mds.a.smithi007.stdout:Jul 20 02:51:19 smithi007 systemd[1]: Starting Ceph mds.a for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:51:19.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:19 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/289027775' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:19.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:19 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/289027775' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:20.629 INFO:journalctl@ceph.mds.a.smithi007.stdout:Jul 20 02:51:20 smithi007 podman[161372]: 2023-07-20T02:51:20.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:20 smithi158 ceph-mon[128925]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:20.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:20 smithi007 ceph-mon[126353]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:20.949 INFO:journalctl@ceph.mds.a.smithi007.stdout:Jul 20 02:51:20 smithi007 bash[161372]: 08cbac41d2896c5ddce3c278fce3f8bb6d05fcf0f34ccb893fd1102aa35a035b 2023-07-20T02:51:20.949 INFO:journalctl@ceph.mds.a.smithi007.stdout:Jul 20 02:51:20 smithi007 ceph-mds[161412]: -- 172.21.15.7:0/1653266298 <== mon.0 v2:172.21.15.7:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c94aba91e0 con 0x55c94ab7f000 2023-07-20T02:51:20.949 INFO:journalctl@ceph.mds.a.smithi007.stdout:Jul 20 02:51:20 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-a[161388]: starting mds.a at 2023-07-20T02:51:20.949 INFO:journalctl@ceph.mds.a.smithi007.stdout:Jul 20 02:51:20 smithi007 systemd[1]: Started Ceph mds.a for 1b04643c-26a7-11ee-9b34-001a4aab830c. 2023-07-20T02:51:20.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:20 smithi188 ceph-mon[129553]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:21.166 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T02:51:21.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.367+0000 7fd5f9fff700 1 -- 172.21.15.7:0/861565471 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f4102300 msgr2=0x7fd5f4108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:21.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.367+0000 7fd5f9fff700 1 --2- 172.21.15.7:0/861565471 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f4102300 0x7fd5f4108bb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec00b0d0 tx=0x7fd5ec00b3e0 comp rx=0 tx=0).stop 2023-07-20T02:51:21.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.367+0000 7fd5f9fff700 1 -- 172.21.15.7:0/861565471 shutdown_connections 2023-07-20T02:51:21.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.367+0000 7fd5f9fff700 1 --2- 172.21.15.7:0/861565471 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f4102300 0x7fd5f4108bb0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:21.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.367+0000 7fd5f9fff700 1 --2- 172.21.15.7:0/861565471 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5f4100770 0x7fd5f4100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:21.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.367+0000 7fd5f9fff700 1 --2- 172.21.15.7:0/861565471 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5f4101960 0x7fd5f4101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:21.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.367+0000 7fd5f9fff700 1 -- 172.21.15.7:0/861565471 >> 172.21.15.7:0/861565471 conn(0x7fd5f40fbca0 msgr2=0x7fd5f40fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:21.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.367+0000 7fd5f9fff700 1 -- 172.21.15.7:0/861565471 shutdown_connections 2023-07-20T02:51:21.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.367+0000 7fd5f9fff700 1 -- 172.21.15.7:0/861565471 wait complete. 2023-07-20T02:51:21.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.368+0000 7fd5f9fff700 1 Processor -- start 2023-07-20T02:51:21.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.368+0000 7fd5f9fff700 1 -- start start 2023-07-20T02:51:21.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.368+0000 7fd5f9fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5f4100770 0x7fd5f4074520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:21.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.369+0000 7fd5f9fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f4101960 0x7fd5f4072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:21.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.369+0000 7fd5f9fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5f4102300 0x7fd5f40730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:21.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.369+0000 7fd5f9fff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f4073840 con 0x7fd5f4101960 2023-07-20T02:51:21.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.369+0000 7fd5f9fff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f40739b0 con 0x7fd5f4102300 2023-07-20T02:51:21.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.369+0000 7fd5f9fff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f41a0520 con 0x7fd5f4100770 2023-07-20T02:51:21.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.369+0000 7fd5f2ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f4101960 0x7fd5f4072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:21.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.369+0000 7fd5f2ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f4101960 0x7fd5f4072b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43830/0 (socket says 172.21.15.7:43830) 2023-07-20T02:51:21.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.369+0000 7fd5f2ffd700 1 -- 172.21.15.7:0/1709096160 learned_addr learned my addr 172.21.15.7:0/1709096160 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:21.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.369+0000 7fd5f37fe700 1 --2- 172.21.15.7:0/1709096160 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5f4100770 0x7fd5f4074520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:21.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.369+0000 7fd5f3fff700 1 --2- 172.21.15.7:0/1709096160 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5f4102300 0x7fd5f40730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:21.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.369+0000 7fd5f2ffd700 1 -- 172.21.15.7:0/1709096160 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5f4100770 msgr2=0x7fd5f4074520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:21.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.369+0000 7fd5f2ffd700 1 --2- 172.21.15.7:0/1709096160 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5f4100770 0x7fd5f4074520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:21.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.369+0000 7fd5f2ffd700 1 -- 172.21.15.7:0/1709096160 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5f4102300 msgr2=0x7fd5f40730a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:21.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.369+0000 7fd5f2ffd700 1 --2- 172.21.15.7:0/1709096160 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5f4102300 0x7fd5f40730a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:21.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.369+0000 7fd5f2ffd700 1 -- 172.21.15.7:0/1709096160 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5ec009d00 con 0x7fd5f4101960 2023-07-20T02:51:21.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.369+0000 7fd5f2ffd700 1 --2- 172.21.15.7:0/1709096160 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f4101960 0x7fd5f4072b60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd5e800c9b0 tx=0x7fd5e800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:21.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.370+0000 7fd5f0ff9700 1 -- 172.21.15.7:0/1709096160 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e8007b00 con 0x7fd5f4101960 2023-07-20T02:51:21.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.370+0000 7fd5f0ff9700 1 -- 172.21.15.7:0/1709096160 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd5e800ce50 con 0x7fd5f4101960 2023-07-20T02:51:21.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.370+0000 7fd5f9fff700 1 -- 172.21.15.7:0/1709096160 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5f41a0800 con 0x7fd5f4101960 2023-07-20T02:51:21.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.370+0000 7fd5f0ff9700 1 -- 172.21.15.7:0/1709096160 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e8018660 con 0x7fd5f4101960 2023-07-20T02:51:21.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.370+0000 7fd5f9fff700 1 -- 172.21.15.7:0/1709096160 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5f41a0d60 con 0x7fd5f4101960 2023-07-20T02:51:21.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.371+0000 7fd5f9fff700 1 -- 172.21.15.7:0/1709096160 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5f410bfe0 con 0x7fd5f4101960 2023-07-20T02:51:21.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.373+0000 7fd5f0ff9700 1 -- 172.21.15.7:0/1709096160 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fd5e800f450 con 0x7fd5f4101960 2023-07-20T02:51:21.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.374+0000 7fd5f0ff9700 1 --2- 172.21.15.7:0/1709096160 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fd5dc0a1220 0x7fd5dc0a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:21.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.374+0000 7fd5f0ff9700 1 -- 172.21.15.7:0/1709096160 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd5e80cc340 con 0x7fd5f4101960 2023-07-20T02:51:21.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.374+0000 7fd5f37fe700 1 --2- 172.21.15.7:0/1709096160 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fd5dc0a1220 0x7fd5dc0a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:21.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.375+0000 7fd5f37fe700 1 --2- 172.21.15.7:0/1709096160 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fd5dc0a1220 0x7fd5dc0a36e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd5f41017c0 tx=0x7fd5e4007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:21.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.378+0000 7fd5f0ff9700 1 -- 172.21.15.7:0/1709096160 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd5e8095e90 con 0x7fd5f4101960 2023-07-20T02:51:21.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.636+0000 7fd5f9fff700 1 -- 172.21.15.7:0/1709096160 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5f410c1a0 con 0x7fd5f4101960 2023-07-20T02:51:21.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.637+0000 7fd5f0ff9700 1 -- 172.21.15.7:0/1709096160 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+8825 (secure 0 0 0) 0x7fd5e8007e60 con 0x7fd5f4101960 2023-07-20T02:51:21.650 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:51:21.650 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/190105946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":190105946},{"type":"v1","addr":"172.21.15.7:6837","nonce":190105946}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/3310846795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":3310846795},{"type":"v1","addr":"172.21.15.7:6839","nonce":3310846795}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14568,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3189534126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3189534126},{"type":"v1","addr":"172.21.15.7:6841","nonce":3189534126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/1435841351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":1435841351},{"type":"v1","addr":"172.21.15.188:6833","nonce":1435841351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/401861265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":401861265},{"type":"v1","addr":"172.21.15.158:6833","nonce":401861265}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/4064058311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":4064058311},{"type":"v1","addr":"172.21.15.7:6835","nonce":4064058311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3931487788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3931487788},{"type":"v1","addr":"172.21.15.158:6835","nonce":3931487788}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/857155289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":857155289},{"type":"v1","addr":"172.21.15.188:6835","nonce":857155289}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1032497949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6836","nonce":1032497949},{"type":"v1","addr":"172.21.15.158:6837","nonce":1032497949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/775436728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":775436728},{"type":"v1","addr":"172.21.15.188:6837","nonce":775436728}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24562,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6839/450074084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":450074084},{"type":"v1","addr":"172.21.15.188:6839","nonce":450074084}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-20T02:51:21.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.640+0000 7fd5f9fff700 1 -- 172.21.15.7:0/1709096160 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fd5dc0a1220 msgr2=0x7fd5dc0a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:21.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.640+0000 7fd5f9fff700 1 --2- 172.21.15.7:0/1709096160 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fd5dc0a1220 0x7fd5dc0a36e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd5f41017c0 tx=0x7fd5e4007400 comp rx=0 tx=0).stop 2023-07-20T02:51:21.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.640+0000 7fd5f9fff700 1 -- 172.21.15.7:0/1709096160 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f4101960 msgr2=0x7fd5f4072b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:21.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.640+0000 7fd5f9fff700 1 --2- 172.21.15.7:0/1709096160 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f4101960 0x7fd5f4072b60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd5e800c9b0 tx=0x7fd5e800ccc0 comp rx=0 tx=0).stop 2023-07-20T02:51:21.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.640+0000 7fd5f9fff700 1 -- 172.21.15.7:0/1709096160 shutdown_connections 2023-07-20T02:51:21.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.640+0000 7fd5f9fff700 1 --2- 172.21.15.7:0/1709096160 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fd5f4102300 0x7fd5f40730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:21.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.640+0000 7fd5f9fff700 1 --2- 172.21.15.7:0/1709096160 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5f4100770 0x7fd5f4074520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:21.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.640+0000 7fd5f9fff700 1 --2- 172.21.15.7:0/1709096160 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fd5dc0a1220 0x7fd5dc0a36e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:21.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.640+0000 7fd5f9fff700 1 --2- 172.21.15.7:0/1709096160 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f4101960 0x7fd5f4072b60 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:21.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.640+0000 7fd5f9fff700 1 -- 172.21.15.7:0/1709096160 >> 172.21.15.7:0/1709096160 conn(0x7fd5f40fbca0 msgr2=0x7fd5f40fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:21.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.640+0000 7fd5f9fff700 1 -- 172.21.15.7:0/1709096160 shutdown_connections 2023-07-20T02:51:21.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:21.640+0000 7fd5f9fff700 1 -- 172.21.15.7:0/1709096160 wait complete. 2023-07-20T02:51:21.654 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 12 2023-07-20T02:51:21.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:21 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:21.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:21 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:21.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:21 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:21.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:21 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:51:21.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:21 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:51:21.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:21 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:51:21.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:21 smithi007 ceph-mon[126353]: Deploying daemon mds.b on smithi158 2023-07-20T02:51:21.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:21 smithi007 ceph-mon[126353]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:21.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:21 smithi007 ceph-mon[126353]: mds.? [v2:172.21.15.7:6840/3189534126,v1:172.21.15.7:6841/3189534126] up:boot 2023-07-20T02:51:21.950 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:21 smithi007 ceph-mon[126353]: fsmap 11 up:standby 2023-07-20T02:51:21.950 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:21 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-20T02:51:21.950 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:21 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/1709096160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:22.090 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:21 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:22.091 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:21 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:22.091 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:21 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:22.091 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:21 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:51:22.091 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:21 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:51:22.091 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:21 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:51:22.091 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:21 smithi158 ceph-mon[128925]: Deploying daemon mds.b on smithi158 2023-07-20T02:51:22.092 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:21 smithi158 ceph-mon[128925]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:22.092 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:21 smithi158 ceph-mon[128925]: mds.? [v2:172.21.15.7:6840/3189534126,v1:172.21.15.7:6841/3189534126] up:boot 2023-07-20T02:51:22.092 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:21 smithi158 ceph-mon[128925]: fsmap 11 up:standby 2023-07-20T02:51:22.092 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:21 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-20T02:51:22.092 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:21 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/1709096160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:22.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:21 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:22.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:21 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:22.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:21 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:22.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:21 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T02:51:22.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:21 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T02:51:22.215 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:21 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:51:22.215 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:21 smithi188 ceph-mon[129553]: Deploying daemon mds.b on smithi158 2023-07-20T02:51:22.215 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:21 smithi188 ceph-mon[129553]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:22.215 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:21 smithi188 ceph-mon[129553]: mds.? [v2:172.21.15.7:6840/3189534126,v1:172.21.15.7:6841/3189534126] up:boot 2023-07-20T02:51:22.215 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:21 smithi188 ceph-mon[129553]: fsmap 11 up:standby 2023-07-20T02:51:22.215 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:21 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-20T02:51:22.215 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:21 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/1709096160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:22.712 INFO:journalctl@ceph.mds.b.smithi158.stdout:Jul 20 02:51:22 smithi158 systemd[1]: Starting Ceph mds.b for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:51:23.009 INFO:journalctl@ceph.mds.b.smithi158.stdout:Jul 20 02:51:22 smithi158 podman[154209]: 2023-07-20T02:51:23.010 INFO:journalctl@ceph.mds.b.smithi158.stdout:Jul 20 02:51:22 smithi158 ceph-mds[154249]: -- 172.21.15.158:0/1217437371 <== mon.2 v2:172.21.15.158:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5631a37bb1e0 con 0x5631a3790800 2023-07-20T02:51:23.010 INFO:journalctl@ceph.mds.b.smithi158.stdout:Jul 20 02:51:22 smithi158 bash[154209]: 7e2b7b1576c5d273ed27e66fdfe67a7d7819e284b5a77c3345d5c26095aaf281 2023-07-20T02:51:23.010 INFO:journalctl@ceph.mds.b.smithi158.stdout:Jul 20 02:51:22 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-b[154225]: starting mds.b at 2023-07-20T02:51:23.390 INFO:journalctl@ceph.mds.b.smithi158.stdout:Jul 20 02:51:22 smithi158 systemd[1]: Started Ceph mds.b for 1b04643c-26a7-11ee-9b34-001a4aab830c. 2023-07-20T02:51:23.673 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T02:51:23.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.879+0000 7f530fdff700 1 -- 172.21.15.7:0/2839038040 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5308102300 msgr2=0x7f5308108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:23.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.879+0000 7f530fdff700 1 --2- 172.21.15.7:0/2839038040 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5308102300 0x7f5308108bb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f530000b0d0 tx=0x7f530000b3e0 comp rx=0 tx=0).stop 2023-07-20T02:51:23.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.879+0000 7f530fdff700 1 -- 172.21.15.7:0/2839038040 shutdown_connections 2023-07-20T02:51:23.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.879+0000 7f530fdff700 1 --2- 172.21.15.7:0/2839038040 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5308102300 0x7f5308108bb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:23.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.879+0000 7f530fdff700 1 --2- 172.21.15.7:0/2839038040 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5308100770 0x7f5308100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:23.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.879+0000 7f530fdff700 1 --2- 172.21.15.7:0/2839038040 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5308101960 0x7f5308101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:23.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.879+0000 7f530fdff700 1 -- 172.21.15.7:0/2839038040 >> 172.21.15.7:0/2839038040 conn(0x7f53080fbca0 msgr2=0x7f53080fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:23.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.879+0000 7f530fdff700 1 -- 172.21.15.7:0/2839038040 shutdown_connections 2023-07-20T02:51:23.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.879+0000 7f530fdff700 1 -- 172.21.15.7:0/2839038040 wait complete. 2023-07-20T02:51:23.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.881+0000 7f530fdff700 1 Processor -- start 2023-07-20T02:51:23.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.881+0000 7f530fdff700 1 -- start start 2023-07-20T02:51:23.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.882+0000 7f530fdff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5308100770 0x7f5308198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:23.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.882+0000 7f530fdff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5308101960 0x7f5308198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:23.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.882+0000 7f530fdff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5308102300 0x7f530819fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:23.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.882+0000 7f530fdff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53081a02c0 con 0x7f5308101960 2023-07-20T02:51:23.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.882+0000 7f530fdff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53081a0400 con 0x7f5308102300 2023-07-20T02:51:23.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.882+0000 7f530fdff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53081a0590 con 0x7f5308100770 2023-07-20T02:51:23.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.882+0000 7f530d39a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5308101960 0x7f5308198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:23.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.882+0000 7f530db9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5308100770 0x7f5308198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:23.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.882+0000 7f530d39a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5308101960 0x7f5308198c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60558/0 (socket says 172.21.15.7:60558) 2023-07-20T02:51:23.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.882+0000 7f530d39a700 1 -- 172.21.15.7:0/1393541602 learned_addr learned my addr 172.21.15.7:0/1393541602 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:23.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.882+0000 7f530e39c700 1 --2- 172.21.15.7:0/1393541602 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5308102300 0x7f530819fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:23.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.882+0000 7f530db9b700 1 -- 172.21.15.7:0/1393541602 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5308102300 msgr2=0x7f530819fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:23.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.882+0000 7f530db9b700 1 --2- 172.21.15.7:0/1393541602 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5308102300 0x7f530819fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:23.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.882+0000 7f530db9b700 1 -- 172.21.15.7:0/1393541602 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5308101960 msgr2=0x7f5308198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:23.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.882+0000 7f530db9b700 1 --2- 172.21.15.7:0/1393541602 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5308101960 0x7f5308198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:23.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.882+0000 7f530db9b700 1 -- 172.21.15.7:0/1393541602 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5300009d00 con 0x7f5308100770 2023-07-20T02:51:23.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.882+0000 7f530d39a700 1 --2- 172.21.15.7:0/1393541602 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5308101960 0x7f5308198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T02:51:23.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.883+0000 7f530db9b700 1 --2- 172.21.15.7:0/1393541602 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5308100770 0x7f5308198740 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f530400e9c0 tx=0x7f530400ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:23.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.883+0000 7f52feffd700 1 -- 172.21.15.7:0/1393541602 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5304004220 con 0x7f5308100770 2023-07-20T02:51:23.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.883+0000 7f530fdff700 1 -- 172.21.15.7:0/1393541602 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53081a0870 con 0x7f5308100770 2023-07-20T02:51:23.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.883+0000 7f52feffd700 1 -- 172.21.15.7:0/1393541602 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5304004380 con 0x7f5308100770 2023-07-20T02:51:23.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.883+0000 7f52feffd700 1 -- 172.21.15.7:0/1393541602 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5304003be0 con 0x7f5308100770 2023-07-20T02:51:23.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.883+0000 7f530fdff700 1 -- 172.21.15.7:0/1393541602 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53081a0d80 con 0x7f5308100770 2023-07-20T02:51:23.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.884+0000 7f530fdff700 1 -- 172.21.15.7:0/1393541602 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f530810bfe0 con 0x7f5308100770 2023-07-20T02:51:23.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.886+0000 7f52feffd700 1 -- 172.21.15.7:0/1393541602 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f5304003d40 con 0x7f5308100770 2023-07-20T02:51:23.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.889+0000 7f52feffd700 1 --2- 172.21.15.7:0/1393541602 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f52f40a11a0 0x7f52f40a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:23.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.889+0000 7f52feffd700 1 -- 172.21.15.7:0/1393541602 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f5304014070 con 0x7f5308100770 2023-07-20T02:51:23.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.889+0000 7f530d39a700 1 --2- 172.21.15.7:0/1393541602 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f52f40a11a0 0x7f52f40a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:23.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.889+0000 7f530d39a700 1 --2- 172.21.15.7:0/1393541602 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f52f40a11a0 0x7f52f40a3660 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f52f8011550 tx=0x7f52f8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:23.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:23.891+0000 7f52feffd700 1 -- 172.21.15.7:0/1393541602 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f53040962f0 con 0x7f5308100770 2023-07-20T02:51:24.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.166+0000 7f530fdff700 1 -- 172.21.15.7:0/1393541602 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f530810c1d0 con 0x7f5308100770 2023-07-20T02:51:24.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.167+0000 7f52feffd700 1 -- 172.21.15.7:0/1393541602 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9586 (secure 0 0 0) 0x7f530810c1d0 con 0x7f5308100770 2023-07-20T02:51:24.180 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:51:24.180 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/190105946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":190105946},{"type":"v1","addr":"172.21.15.7:6837","nonce":190105946}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/3310846795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":3310846795},{"type":"v1","addr":"172.21.15.7:6839","nonce":3310846795}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14568,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3189534126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3189534126},{"type":"v1","addr":"172.21.15.7:6841","nonce":3189534126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/1435841351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":1435841351},{"type":"v1","addr":"172.21.15.188:6833","nonce":1435841351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/401861265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":401861265},{"type":"v1","addr":"172.21.15.158:6833","nonce":401861265}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/4064058311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":4064058311},{"type":"v1","addr":"172.21.15.7:6835","nonce":4064058311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3931487788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3931487788},{"type":"v1","addr":"172.21.15.158:6835","nonce":3931487788}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/857155289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":857155289},{"type":"v1","addr":"172.21.15.188:6835","nonce":857155289}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1032497949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6836","nonce":1032497949},{"type":"v1","addr":"172.21.15.158:6837","nonce":1032497949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/775436728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":775436728},{"type":"v1","addr":"172.21.15.188:6837","nonce":775436728}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24562,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6839/450074084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":450074084},{"type":"v1","addr":"172.21.15.188:6839","nonce":450074084}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24572,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6839/2552719389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6838","nonce":2552719389},{"type":"v1","addr":"172.21.15.158:6839","nonce":2552719389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[]} 2023-07-20T02:51:24.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.170+0000 7f530fdff700 1 -- 172.21.15.7:0/1393541602 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f52f40a11a0 msgr2=0x7f52f40a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:24.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.170+0000 7f530fdff700 1 --2- 172.21.15.7:0/1393541602 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f52f40a11a0 0x7f52f40a3660 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f52f8011550 tx=0x7f52f8009260 comp rx=0 tx=0).stop 2023-07-20T02:51:24.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.170+0000 7f530fdff700 1 -- 172.21.15.7:0/1393541602 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5308100770 msgr2=0x7f5308198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:24.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.170+0000 7f530fdff700 1 --2- 172.21.15.7:0/1393541602 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5308100770 0x7f5308198740 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f530400e9c0 tx=0x7f530400ecd0 comp rx=0 tx=0).stop 2023-07-20T02:51:24.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.170+0000 7f530fdff700 1 -- 172.21.15.7:0/1393541602 shutdown_connections 2023-07-20T02:51:24.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.170+0000 7f530fdff700 1 --2- 172.21.15.7:0/1393541602 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f5308102300 0x7f530819fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:24.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.170+0000 7f530fdff700 1 --2- 172.21.15.7:0/1393541602 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5308100770 0x7f5308198740 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:24.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.170+0000 7f530fdff700 1 --2- 172.21.15.7:0/1393541602 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f52f40a11a0 0x7f52f40a3660 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:24.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.170+0000 7f530fdff700 1 --2- 172.21.15.7:0/1393541602 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5308101960 0x7f5308198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:24.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.170+0000 7f530fdff700 1 -- 172.21.15.7:0/1393541602 >> 172.21.15.7:0/1393541602 conn(0x7f53080fbca0 msgr2=0x7f53081071c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:24.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.170+0000 7f530fdff700 1 -- 172.21.15.7:0/1393541602 shutdown_connections 2023-07-20T02:51:24.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.170+0000 7f530fdff700 1 -- 172.21.15.7:0/1393541602 wait complete. 2023-07-20T02:51:24.185 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 13 2023-07-20T02:51:24.202 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T02:51:24.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:24 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:24.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:24 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:24.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:24 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:24.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:24 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:24.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:24 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:24.392 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:24 smithi158 ceph-mon[128925]: mds.? [v2:172.21.15.158:6838/2552719389,v1:172.21.15.158:6839/2552719389] up:boot 2023-07-20T02:51:24.392 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:24 smithi158 ceph-mon[128925]: fsmap 12 up:standby 2023-07-20T02:51:24.392 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:24 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-20T02:51:24.392 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:24 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:51:24.392 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:24 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:24.392 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:24 smithi158 ceph-mon[128925]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:24.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.402+0000 7fa8795ff700 1 -- 172.21.15.7:0/2949448975 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa874101960 msgr2=0x7fa874101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:24.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.402+0000 7fa8795ff700 1 --2- 172.21.15.7:0/2949448975 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa874101960 0x7fa874101dc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa85c009920 tx=0x7fa85c009c30 comp rx=0 tx=0).stop 2023-07-20T02:51:24.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.402+0000 7fa8795ff700 1 -- 172.21.15.7:0/2949448975 shutdown_connections 2023-07-20T02:51:24.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.402+0000 7fa8795ff700 1 --2- 172.21.15.7:0/2949448975 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa874102300 0x7fa874108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:24.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.402+0000 7fa8795ff700 1 --2- 172.21.15.7:0/2949448975 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa874100770 0x7fa874100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:24.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.402+0000 7fa8795ff700 1 --2- 172.21.15.7:0/2949448975 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa874101960 0x7fa874101dc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:24.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.402+0000 7fa8795ff700 1 -- 172.21.15.7:0/2949448975 >> 172.21.15.7:0/2949448975 conn(0x7fa8740fbca0 msgr2=0x7fa8740fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:24.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.402+0000 7fa8795ff700 1 -- 172.21.15.7:0/2949448975 shutdown_connections 2023-07-20T02:51:24.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.402+0000 7fa8795ff700 1 -- 172.21.15.7:0/2949448975 wait complete. 2023-07-20T02:51:24.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.404+0000 7fa8795ff700 1 Processor -- start 2023-07-20T02:51:24.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.404+0000 7fa8795ff700 1 -- start start 2023-07-20T02:51:24.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.404+0000 7fa8795ff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa874100770 0x7fa874074520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:24.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.404+0000 7fa8795ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa874101960 0x7fa874072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:24.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.404+0000 7fa8795ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa874102300 0x7fa8740730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:24.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.404+0000 7fa8795ff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa874073840 con 0x7fa874101960 2023-07-20T02:51:24.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.404+0000 7fa8795ff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8740739b0 con 0x7fa874100770 2023-07-20T02:51:24.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.404+0000 7fa8795ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8741a0560 con 0x7fa874102300 2023-07-20T02:51:24.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.405+0000 7fa872ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa874100770 0x7fa874074520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:24.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.405+0000 7fa872ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa874100770 0x7fa874074520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.7:55826/0 (socket says 172.21.15.7:55826) 2023-07-20T02:51:24.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.405+0000 7fa872ffd700 1 -- 172.21.15.7:0/2648641245 learned_addr learned my addr 172.21.15.7:0/2648641245 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:24.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.405+0000 7fa8737fe700 1 --2- 172.21.15.7:0/2648641245 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa874102300 0x7fa8740730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:24.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.405+0000 7fa86bfff700 1 --2- 172.21.15.7:0/2648641245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa874101960 0x7fa874072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:24.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.405+0000 7fa872ffd700 1 -- 172.21.15.7:0/2648641245 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa874102300 msgr2=0x7fa8740730a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:24.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.405+0000 7fa872ffd700 1 --2- 172.21.15.7:0/2648641245 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa874102300 0x7fa8740730a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:24.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.405+0000 7fa872ffd700 1 -- 172.21.15.7:0/2648641245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa874101960 msgr2=0x7fa874072b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:24.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.405+0000 7fa872ffd700 1 --2- 172.21.15.7:0/2648641245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa874101960 0x7fa874072b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:24.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.405+0000 7fa872ffd700 1 -- 172.21.15.7:0/2648641245 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa85c009520 con 0x7fa874100770 2023-07-20T02:51:24.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.405+0000 7fa872ffd700 1 --2- 172.21.15.7:0/2648641245 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa874100770 0x7fa874074520 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa86400e9f0 tx=0x7fa86400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:24.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.405+0000 7fa870ff9700 1 -- 172.21.15.7:0/2648641245 <== mon.2 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa864004220 con 0x7fa874100770 2023-07-20T02:51:24.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.405+0000 7fa870ff9700 1 -- 172.21.15.7:0/2648641245 <== mon.2 v2:172.21.15.158:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa864004380 con 0x7fa874100770 2023-07-20T02:51:24.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.405+0000 7fa8795ff700 1 -- 172.21.15.7:0/2648641245 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8741a0840 con 0x7fa874100770 2023-07-20T02:51:24.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.406+0000 7fa870ff9700 1 -- 172.21.15.7:0/2648641245 <== mon.2 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa864003be0 con 0x7fa874100770 2023-07-20T02:51:24.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.406+0000 7fa8795ff700 1 -- 172.21.15.7:0/2648641245 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8741a0d50 con 0x7fa874100770 2023-07-20T02:51:24.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.407+0000 7fa8795ff700 1 -- 172.21.15.7:0/2648641245 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa87410bfe0 con 0x7fa874100770 2023-07-20T02:51:24.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.409+0000 7fa870ff9700 1 -- 172.21.15.7:0/2648641245 <== mon.2 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fa864003d40 con 0x7fa874100770 2023-07-20T02:51:24.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.411+0000 7fa870ff9700 1 --2- 172.21.15.7:0/2648641245 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fa8600a11a0 0x7fa8600a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:24.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.411+0000 7fa870ff9700 1 -- 172.21.15.7:0/2648641245 <== mon.2 v2:172.21.15.158:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa864014070 con 0x7fa874100770 2023-07-20T02:51:24.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.411+0000 7fa86bfff700 1 --2- 172.21.15.7:0/2648641245 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fa8600a11a0 0x7fa8600a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:24.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.412+0000 7fa86bfff700 1 --2- 172.21.15.7:0/2648641245 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fa8600a11a0 0x7fa8600a3660 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa85c00fa40 tx=0x7fa85c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:24.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.412+0000 7fa870ff9700 1 -- 172.21.15.7:0/2648641245 <== mon.2 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa864096440 con 0x7fa874100770 2023-07-20T02:51:24.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:24 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:24.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:24 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:24.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:24 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:24.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:24 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:24.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:24 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:24.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:24 smithi007 ceph-mon[126353]: mds.? [v2:172.21.15.158:6838/2552719389,v1:172.21.15.158:6839/2552719389] up:boot 2023-07-20T02:51:24.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:24 smithi007 ceph-mon[126353]: fsmap 12 up:standby 2023-07-20T02:51:24.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:24 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-20T02:51:24.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:24 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:51:24.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:24 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:24.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:24 smithi007 ceph-mon[126353]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:24.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:24 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:24.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:24 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:24.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:24 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:24.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:24 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:24.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:24 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:24.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:24 smithi188 ceph-mon[129553]: mds.? [v2:172.21.15.158:6838/2552719389,v1:172.21.15.158:6839/2552719389] up:boot 2023-07-20T02:51:24.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:24 smithi188 ceph-mon[129553]: fsmap 12 up:standby 2023-07-20T02:51:24.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:24 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-20T02:51:24.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:24 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:51:24.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:24 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:24.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:24 smithi188 ceph-mon[129553]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:24.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.640+0000 7fa8795ff700 1 -- 172.21.15.7:0/2648641245 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa87410c200 con 0x7fa874100770 2023-07-20T02:51:24.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.641+0000 7fa870ff9700 1 -- 172.21.15.7:0/2648641245 <== mon.2 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+18389 (secure 0 0 0) 0x7fa864099a60 con 0x7fa874100770 2023-07-20T02:51:24.654 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:51:24.654 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":79,"fsid":"1b04643c-26a7-11ee-9b34-001a4aab830c","created":"2023-07-20T02:43:56.544719+0000","modified":"2023-07-20T02:50:41.133457+0000","last_up_change":"2023-07-20T02:50:40.126926+0000","last_in_change":"2023-07-20T02:50:21.875020+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T02:47:12.055539+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad6c6339-6e23-4c21-9fde-2dbecdafdb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6803","nonce":3322992091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6805","nonce":3322992091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6809","nonce":3322992091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6807","nonce":3322992091}]},"public_addr":"172.21.15.7:6803/3322992091","cluster_addr":"172.21.15.7:6805/3322992091","heartbeat_back_addr":"172.21.15.7:6809/3322992091","heartbeat_front_addr":"172.21.15.7:6807/3322992091","state":["exists","up"]},{"osd":1,"uuid":"cad207b3-a1ea-4f07-bd94-9b11fd573a69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6801","nonce":2796771612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6803","nonce":2796771612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6807","nonce":2796771612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6805","nonce":2796771612}]},"public_addr":"172.21.15.158:6801/2796771612","cluster_addr":"172.21.15.158:6803/2796771612","heartbeat_back_addr":"172.21.15.158:6807/2796771612","heartbeat_front_addr":"172.21.15.158:6805/2796771612","state":["exists","up"]},{"osd":2,"uuid":"2c777fe6-d003-49d0-af37-f63fca37ff06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6801","nonce":2928307258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6803","nonce":2928307258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6807","nonce":2928307258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6805","nonce":2928307258}]},"public_addr":"172.21.15.188:6801/2928307258","cluster_addr":"172.21.15.188:6803/2928307258","heartbeat_back_addr":"172.21.15.188:6807/2928307258","heartbeat_front_addr":"172.21.15.188:6805/2928307258","state":["exists","up"]},{"osd":3,"uuid":"59e29965-018f-4eb6-adbd-1a9e1df84245","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6811","nonce":2192953827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6813","nonce":2192953827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6817","nonce":2192953827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6815","nonce":2192953827}]},"public_addr":"172.21.15.7:6811/2192953827","cluster_addr":"172.21.15.7:6813/2192953827","heartbeat_back_addr":"172.21.15.7:6817/2192953827","heartbeat_front_addr":"172.21.15.7:6815/2192953827","state":["exists","up"]},{"osd":4,"uuid":"e2453384-3b09-4f2a-bea7-172d8fc466d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6809","nonce":3330591132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6810","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6811","nonce":3330591132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6814","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6815","nonce":3330591132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6813","nonce":3330591132}]},"public_addr":"172.21.15.158:6809/3330591132","cluster_addr":"172.21.15.158:6811/3330591132","heartbeat_back_addr":"172.21.15.158:6815/3330591132","heartbeat_front_addr":"172.21.15.158:6813/3330591132","state":["exists","up"]},{"osd":5,"uuid":"c3c04c13-0ce5-440d-9e62-0e301ad9f771","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6809","nonce":3142012805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6811","nonce":3142012805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6815","nonce":3142012805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6813","nonce":3142012805}]},"public_addr":"172.21.15.188:6809/3142012805","cluster_addr":"172.21.15.188:6811/3142012805","heartbeat_back_addr":"172.21.15.188:6815/3142012805","heartbeat_front_addr":"172.21.15.188:6813/3142012805","state":["exists","up"]},{"osd":6,"uuid":"82ef2d33-1045-41f1-bf69-b03404bcace8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6819","nonce":859560576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6821","nonce":859560576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6825","nonce":859560576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6823","nonce":859560576}]},"public_addr":"172.21.15.7:6819/859560576","cluster_addr":"172.21.15.7:6821/859560576","heartbeat_back_addr":"172.21.15.7:6825/859560576","heartbeat_front_addr":"172.21.15.7:6823/859560576","state":["exists","up"]},{"osd":7,"uuid":"020ec859-3fc2-4da2-a0dc-10530a4a366c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6817","nonce":2578758635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6819","nonce":2578758635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6822","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6823","nonce":2578758635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6821","nonce":2578758635}]},"public_addr":"172.21.15.158:6817/2578758635","cluster_addr":"172.21.15.158:6819/2578758635","heartbeat_back_addr":"172.21.15.158:6823/2578758635","heartbeat_front_addr":"172.21.15.158:6821/2578758635","state":["exists","up"]},{"osd":8,"uuid":"4081496a-45ee-4b25-8f17-a23665521187","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6817","nonce":3020824843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6819","nonce":3020824843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6823","nonce":3020824843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6821","nonce":3020824843}]},"public_addr":"172.21.15.188:6817/3020824843","cluster_addr":"172.21.15.188:6819/3020824843","heartbeat_back_addr":"172.21.15.188:6823/3020824843","heartbeat_front_addr":"172.21.15.188:6821/3020824843","state":["exists","up"]},{"osd":9,"uuid":"723580ae-efe3-4d86-95fd-d7a4c0631425","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6827","nonce":1612464825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6829","nonce":1612464825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6833","nonce":1612464825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6831","nonce":1612464825}]},"public_addr":"172.21.15.7:6827/1612464825","cluster_addr":"172.21.15.7:6829/1612464825","heartbeat_back_addr":"172.21.15.7:6833/1612464825","heartbeat_front_addr":"172.21.15.7:6831/1612464825","state":["exists","up"]},{"osd":10,"uuid":"c2651548-fe46-4d2d-b911-278036412ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6825","nonce":1653166210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6827","nonce":1653166210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6831","nonce":1653166210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6829","nonce":1653166210}]},"public_addr":"172.21.15.158:6825/1653166210","cluster_addr":"172.21.15.158:6827/1653166210","heartbeat_back_addr":"172.21.15.158:6831/1653166210","heartbeat_front_addr":"172.21.15.158:6829/1653166210","state":["exists","up"]},{"osd":11,"uuid":"f7262bcb-4243-403a-a321-fc2bdf1d4d25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6825","nonce":1230196525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6827","nonce":1230196525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6831","nonce":1230196525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6829","nonce":1230196525}]},"public_addr":"172.21.15.188:6825/1230196525","cluster_addr":"172.21.15.188:6827/1230196525","heartbeat_back_addr":"172.21.15.188:6831/1230196525","heartbeat_front_addr":"172.21.15.188:6829/1230196525","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:46:26.803832+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:46:48.726253+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:47:09.744123+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:47:36.907588+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:47:57.696685+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:48:19.120806+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:48:45.313043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:49:06.686384+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:49:28.475078+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:49:54.984033+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:50:16.541259+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:50:38.221746+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/3892248290":"2023-07-21T02:44:53.458649+0000","172.21.15.7:0/2373058722":"2023-07-21T02:44:53.458649+0000","172.21.15.7:6801/2926310425":"2023-07-21T02:44:53.458649+0000","172.21.15.7:0/4222569264":"2023-07-21T02:44:25.804258+0000","172.21.15.7:0/1357065986":"2023-07-21T02:44:25.804258+0000","172.21.15.7:0/884368977":"2023-07-21T02:44:53.458649+0000","172.21.15.7:6801/1224472946":"2023-07-21T02:44:25.804258+0000","172.21.15.7:0/1459293748":"2023-07-21T02:44:25.804258+0000","172.21.15.7:6800/1224472946":"2023-07-21T02:44:25.804258+0000","172.21.15.7:6800/2926310425":"2023-07-21T02:44:53.458649+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T02:51:24.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.644+0000 7fa8795ff700 1 -- 172.21.15.7:0/2648641245 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fa8600a11a0 msgr2=0x7fa8600a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:24.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.644+0000 7fa8795ff700 1 --2- 172.21.15.7:0/2648641245 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fa8600a11a0 0x7fa8600a3660 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa85c00fa40 tx=0x7fa85c019040 comp rx=0 tx=0).stop 2023-07-20T02:51:24.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.644+0000 7fa8795ff700 1 -- 172.21.15.7:0/2648641245 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa874100770 msgr2=0x7fa874074520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:24.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.644+0000 7fa8795ff700 1 --2- 172.21.15.7:0/2648641245 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa874100770 0x7fa874074520 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa86400e9f0 tx=0x7fa86400ed00 comp rx=0 tx=0).stop 2023-07-20T02:51:24.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.644+0000 7fa8795ff700 1 -- 172.21.15.7:0/2648641245 shutdown_connections 2023-07-20T02:51:24.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.644+0000 7fa8795ff700 1 --2- 172.21.15.7:0/2648641245 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa874102300 0x7fa8740730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:24.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.644+0000 7fa8795ff700 1 --2- 172.21.15.7:0/2648641245 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa874100770 0x7fa874074520 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:24.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.644+0000 7fa8795ff700 1 --2- 172.21.15.7:0/2648641245 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fa8600a11a0 0x7fa8600a3660 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:24.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.644+0000 7fa8795ff700 1 --2- 172.21.15.7:0/2648641245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa874101960 0x7fa874072b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:24.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.644+0000 7fa8795ff700 1 -- 172.21.15.7:0/2648641245 >> 172.21.15.7:0/2648641245 conn(0x7fa8740fbca0 msgr2=0x7fa8740fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:24.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.646+0000 7fa8795ff700 1 -- 172.21.15.7:0/2648641245 shutdown_connections 2023-07-20T02:51:24.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.646+0000 7fa8795ff700 1 -- 172.21.15.7:0/2648641245 wait complete. 2023-07-20T02:51:24.678 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-20T02:47:12.055539+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-20T02:51:24.679 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-20T02:51:24.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.882+0000 7faf6d5ff700 1 -- 172.21.15.7:0/848857112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf68073170 msgr2=0x7faf680735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:24.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.882+0000 7faf6d5ff700 1 --2- 172.21.15.7:0/848857112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf68073170 0x7faf680735f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7faf5c0098f0 tx=0x7faf5c009c00 comp rx=0 tx=0).stop 2023-07-20T02:51:24.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.882+0000 7faf6d5ff700 1 -- 172.21.15.7:0/848857112 shutdown_connections 2023-07-20T02:51:24.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.882+0000 7faf6d5ff700 1 --2- 172.21.15.7:0/848857112 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf68108630 0x7faf6810aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:24.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.882+0000 7faf6d5ff700 1 --2- 172.21.15.7:0/848857112 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf68074750 0x7faf68072ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:24.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.882+0000 7faf6d5ff700 1 --2- 172.21.15.7:0/848857112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf68073170 0x7faf680735f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:24.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.882+0000 7faf6d5ff700 1 -- 172.21.15.7:0/848857112 >> 172.21.15.7:0/848857112 conn(0x7faf680fbcb0 msgr2=0x7faf680fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:24.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.882+0000 7faf6d5ff700 1 -- 172.21.15.7:0/848857112 shutdown_connections 2023-07-20T02:51:24.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.882+0000 7faf6d5ff700 1 -- 172.21.15.7:0/848857112 wait complete. 2023-07-20T02:51:24.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.883+0000 7faf6d5ff700 1 Processor -- start 2023-07-20T02:51:24.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.883+0000 7faf6d5ff700 1 -- start start 2023-07-20T02:51:24.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.883+0000 7faf6d5ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf68073170 0x7faf681986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:24.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.883+0000 7faf6d5ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf68074750 0x7faf68198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:24.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.883+0000 7faf6d5ff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf68108630 0x7faf6819fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:24.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.883+0000 7faf6d5ff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf681a0250 con 0x7faf68073170 2023-07-20T02:51:24.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.883+0000 7faf6d5ff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf681a0390 con 0x7faf68108630 2023-07-20T02:51:24.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.883+0000 7faf6d5ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf681a0520 con 0x7faf68074750 2023-07-20T02:51:24.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.884+0000 7faf66ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf68073170 0x7faf681986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:24.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.884+0000 7faf66ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf68073170 0x7faf681986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60592/0 (socket says 172.21.15.7:60592) 2023-07-20T02:51:24.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.884+0000 7faf667fc700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf68074750 0x7faf68198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:24.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.884+0000 7faf66ffd700 1 -- 172.21.15.7:0/116505691 learned_addr learned my addr 172.21.15.7:0/116505691 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:24.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.884+0000 7faf677fe700 1 --2- 172.21.15.7:0/116505691 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf68108630 0x7faf6819fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:24.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.884+0000 7faf66ffd700 1 -- 172.21.15.7:0/116505691 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf68074750 msgr2=0x7faf68198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:24.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.884+0000 7faf66ffd700 1 --2- 172.21.15.7:0/116505691 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf68074750 0x7faf68198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:24.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.884+0000 7faf66ffd700 1 -- 172.21.15.7:0/116505691 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf68108630 msgr2=0x7faf6819fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:24.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.884+0000 7faf66ffd700 1 --2- 172.21.15.7:0/116505691 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf68108630 0x7faf6819fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:24.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.884+0000 7faf66ffd700 1 -- 172.21.15.7:0/116505691 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf60009d00 con 0x7faf68073170 2023-07-20T02:51:24.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.884+0000 7faf667fc700 1 --2- 172.21.15.7:0/116505691 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf68074750 0x7faf68198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T02:51:24.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.884+0000 7faf66ffd700 1 --2- 172.21.15.7:0/116505691 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf68073170 0x7faf681986d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7faf5800e9a0 tx=0x7faf5800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:24.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.884+0000 7faf4ffff700 1 -- 172.21.15.7:0/116505691 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf58009790 con 0x7faf68073170 2023-07-20T02:51:24.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.885+0000 7faf4ffff700 1 -- 172.21.15.7:0/116505691 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faf58010430 con 0x7faf68073170 2023-07-20T02:51:24.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.885+0000 7faf6d5ff700 1 -- 172.21.15.7:0/116505691 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf5c009520 con 0x7faf68073170 2023-07-20T02:51:24.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.885+0000 7faf4ffff700 1 -- 172.21.15.7:0/116505691 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf58018680 con 0x7faf68073170 2023-07-20T02:51:24.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.885+0000 7faf6d5ff700 1 -- 172.21.15.7:0/116505691 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf681a0b80 con 0x7faf68073170 2023-07-20T02:51:24.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.886+0000 7faf6d5ff700 1 -- 172.21.15.7:0/116505691 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf68106d30 con 0x7faf68073170 2023-07-20T02:51:24.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.888+0000 7faf4ffff700 1 -- 172.21.15.7:0/116505691 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7faf580105a0 con 0x7faf68073170 2023-07-20T02:51:24.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.890+0000 7faf4ffff700 1 --2- 172.21.15.7:0/116505691 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7faf500a1150 0x7faf500a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:24.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.890+0000 7faf4ffff700 1 -- 172.21.15.7:0/116505691 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7faf58014070 con 0x7faf68073170 2023-07-20T02:51:24.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.890+0000 7faf667fc700 1 --2- 172.21.15.7:0/116505691 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7faf500a1150 0x7faf500a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:24.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.891+0000 7faf667fc700 1 --2- 172.21.15.7:0/116505691 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7faf500a1150 0x7faf500a3610 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7faf5c005e90 tx=0x7faf5c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:24.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:24.894+0000 7faf4ffff700 1 -- 172.21.15.7:0/116505691 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7faf58051780 con 0x7faf68073170 2023-07-20T02:51:25.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.113+0000 7faf6d5ff700 1 -- 172.21.15.7:0/116505691 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faf68061380 con 0x7faf68073170 2023-07-20T02:51:25.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.114+0000 7faf4ffff700 1 -- 172.21.15.7:0/116505691 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7faf58096220 con 0x7faf68073170 2023-07-20T02:51:25.127 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2023-07-20T02:51:25.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.116+0000 7faf6d5ff700 1 -- 172.21.15.7:0/116505691 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7faf500a1150 msgr2=0x7faf500a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:25.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.116+0000 7faf6d5ff700 1 --2- 172.21.15.7:0/116505691 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7faf500a1150 0x7faf500a3610 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7faf5c005e90 tx=0x7faf5c019040 comp rx=0 tx=0).stop 2023-07-20T02:51:25.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.116+0000 7faf6d5ff700 1 -- 172.21.15.7:0/116505691 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf68073170 msgr2=0x7faf681986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:25.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.116+0000 7faf6d5ff700 1 --2- 172.21.15.7:0/116505691 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf68073170 0x7faf681986d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7faf5800e9a0 tx=0x7faf5800ecb0 comp rx=0 tx=0).stop 2023-07-20T02:51:25.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.116+0000 7faf6d5ff700 1 -- 172.21.15.7:0/116505691 shutdown_connections 2023-07-20T02:51:25.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.116+0000 7faf6d5ff700 1 --2- 172.21.15.7:0/116505691 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7faf68108630 0x7faf6819fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:25.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.116+0000 7faf6d5ff700 1 --2- 172.21.15.7:0/116505691 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7faf500a1150 0x7faf500a3610 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:25.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.116+0000 7faf6d5ff700 1 --2- 172.21.15.7:0/116505691 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf68073170 0x7faf681986d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:25.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.116+0000 7faf6d5ff700 1 --2- 172.21.15.7:0/116505691 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf68074750 0x7faf68198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:25.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.116+0000 7faf6d5ff700 1 -- 172.21.15.7:0/116505691 >> 172.21.15.7:0/116505691 conn(0x7faf680fbcb0 msgr2=0x7faf680fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:25.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.117+0000 7faf6d5ff700 1 -- 172.21.15.7:0/116505691 shutdown_connections 2023-07-20T02:51:25.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.117+0000 7faf6d5ff700 1 -- 172.21.15.7:0/116505691 wait complete. 2023-07-20T02:51:25.155 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-07-20T02:51:25.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.360+0000 7f04b67ff700 1 -- 172.21.15.7:0/3641243210 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f04b0102310 msgr2=0x7f04b0108bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:25.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.360+0000 7f04b67ff700 1 --2- 172.21.15.7:0/3641243210 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f04b0102310 0x7f04b0108bc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f04a8009f90 tx=0x7f04a800b330 comp rx=0 tx=0).stop 2023-07-20T02:51:25.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.360+0000 7f04b67ff700 1 -- 172.21.15.7:0/3641243210 shutdown_connections 2023-07-20T02:51:25.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.360+0000 7f04b67ff700 1 --2- 172.21.15.7:0/3641243210 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f04b0102310 0x7f04b0108bc0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:25.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.360+0000 7f04b67ff700 1 --2- 172.21.15.7:0/3641243210 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04b0100780 0x7f04b0100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:25.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.360+0000 7f04b67ff700 1 --2- 172.21.15.7:0/3641243210 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04b0101970 0x7f04b0101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:25.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.360+0000 7f04b67ff700 1 -- 172.21.15.7:0/3641243210 >> 172.21.15.7:0/3641243210 conn(0x7f04b00fbcb0 msgr2=0x7f04b00fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:25.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.361+0000 7f04b67ff700 1 -- 172.21.15.7:0/3641243210 shutdown_connections 2023-07-20T02:51:25.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.361+0000 7f04b67ff700 1 -- 172.21.15.7:0/3641243210 wait complete. 2023-07-20T02:51:25.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.362+0000 7f04b67ff700 1 Processor -- start 2023-07-20T02:51:25.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.362+0000 7f04b67ff700 1 -- start start 2023-07-20T02:51:25.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.362+0000 7f04b67ff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04b0100780 0x7f04b0198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:25.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.362+0000 7f04b67ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f04b0101970 0x7f04b0198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:25.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.362+0000 7f04b67ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04b0102310 0x7f04b019fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:25.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.362+0000 7f04b67ff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b01a02e0 con 0x7f04b0102310 2023-07-20T02:51:25.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.362+0000 7f04b67ff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b01a0420 con 0x7f04b0100780 2023-07-20T02:51:25.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.362+0000 7f04b67ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b01a05b0 con 0x7f04b0101970 2023-07-20T02:51:25.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.363+0000 7f04af7fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f04b0101970 0x7f04b0198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:25.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.363+0000 7f04affff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04b0100780 0x7f04b0198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:25.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.363+0000 7f04b4d9c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04b0102310 0x7f04b019fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:25.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.363+0000 7f04af7fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f04b0101970 0x7f04b0198ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.7:53968/0 (socket says 172.21.15.7:53968) 2023-07-20T02:51:25.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.363+0000 7f04affff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04b0100780 0x7f04b0198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.7:55850/0 (socket says 172.21.15.7:55850) 2023-07-20T02:51:25.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.363+0000 7f04af7fe700 1 -- 172.21.15.7:0/3850098619 learned_addr learned my addr 172.21.15.7:0/3850098619 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:25.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.363+0000 7f04af7fe700 1 -- 172.21.15.7:0/3850098619 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04b0100780 msgr2=0x7f04b0198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:25.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.363+0000 7f04af7fe700 1 --2- 172.21.15.7:0/3850098619 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04b0100780 0x7f04b0198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:25.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.363+0000 7f04af7fe700 1 -- 172.21.15.7:0/3850098619 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04b0102310 msgr2=0x7f04b019fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:25.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.363+0000 7f04af7fe700 1 --2- 172.21.15.7:0/3850098619 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04b0102310 0x7f04b019fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:25.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.363+0000 7f04af7fe700 1 -- 172.21.15.7:0/3850098619 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a40095f0 con 0x7f04b0101970 2023-07-20T02:51:25.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.363+0000 7f04b4d9c700 1 --2- 172.21.15.7:0/3850098619 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04b0102310 0x7f04b019fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T02:51:25.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.363+0000 7f04af7fe700 1 --2- 172.21.15.7:0/3850098619 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f04b0101970 0x7f04b0198ca0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f04a400ec10 tx=0x7f04a400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:25.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.363+0000 7f04ad7fa700 1 -- 172.21.15.7:0/3850098619 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a400cc90 con 0x7f04b0101970 2023-07-20T02:51:25.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.364+0000 7f04ad7fa700 1 -- 172.21.15.7:0/3850098619 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f04a40044e0 con 0x7f04b0101970 2023-07-20T02:51:25.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.364+0000 7f04ad7fa700 1 -- 172.21.15.7:0/3850098619 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a4010430 con 0x7f04b0101970 2023-07-20T02:51:25.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.364+0000 7f04b67ff700 1 -- 172.21.15.7:0/3850098619 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04a8009c40 con 0x7f04b0101970 2023-07-20T02:51:25.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.364+0000 7f04b67ff700 1 -- 172.21.15.7:0/3850098619 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b01a0c10 con 0x7f04b0101970 2023-07-20T02:51:25.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.365+0000 7f04b67ff700 1 -- 172.21.15.7:0/3850098619 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04b010bff0 con 0x7f04b0101970 2023-07-20T02:51:25.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.368+0000 7f04ad7fa700 1 -- 172.21.15.7:0/3850098619 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f04a4003fb0 con 0x7f04b0101970 2023-07-20T02:51:25.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.370+0000 7f04ad7fa700 1 --2- 172.21.15.7:0/3850098619 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f04980a1220 0x7f04980a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:25.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.370+0000 7f04ad7fa700 1 -- 172.21.15.7:0/3850098619 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f04a4014070 con 0x7f04b0101970 2023-07-20T02:51:25.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.370+0000 7f04affff700 1 --2- 172.21.15.7:0/3850098619 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f04980a1220 0x7f04980a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:25.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.370+0000 7f04ad7fa700 1 -- 172.21.15.7:0/3850098619 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f04a4096b60 con 0x7f04b0101970 2023-07-20T02:51:25.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.371+0000 7f04affff700 1 --2- 172.21.15.7:0/3850098619 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f04980a1220 0x7f04980a36e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f04b01017d0 tx=0x7f04a0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:25.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:25 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/1393541602' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:25.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:25 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/2648641245' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:51:25.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:25 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/1393541602' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:25.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:25 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/2648641245' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:51:25.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:25 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/1393541602' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:25.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:25 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/2648641245' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:51:25.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.606+0000 7f04b67ff700 1 -- 172.21.15.7:0/3850098619 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f04b01a12b0 con 0x7f04b0101970 2023-07-20T02:51:25.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.607+0000 7f04ad7fa700 1 -- 172.21.15.7:0/3850098619 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v79) v1 ==== 84+0+65 (secure 0 0 0) 0x7f04a409a180 con 0x7f04b0101970 2023-07-20T02:51:25.619 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:51:25.619 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2023-07-20T02:51:25.620 INFO:teuthology.orchestra.run.smithi007.stdout: { 2023-07-20T02:51:25.620 INFO:teuthology.orchestra.run.smithi007.stdout: "poolnum": 1, 2023-07-20T02:51:25.620 INFO:teuthology.orchestra.run.smithi007.stdout: "poolname": ".mgr" 2023-07-20T02:51:25.620 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-07-20T02:51:25.620 INFO:teuthology.orchestra.run.smithi007.stdout:] 2023-07-20T02:51:25.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.609+0000 7f04b67ff700 1 -- 172.21.15.7:0/3850098619 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f04980a1220 msgr2=0x7f04980a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:25.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.609+0000 7f04b67ff700 1 --2- 172.21.15.7:0/3850098619 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f04980a1220 0x7f04980a36e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f04b01017d0 tx=0x7f04a0009260 comp rx=0 tx=0).stop 2023-07-20T02:51:25.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.609+0000 7f04b67ff700 1 -- 172.21.15.7:0/3850098619 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f04b0101970 msgr2=0x7f04b0198ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:25.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.609+0000 7f04b67ff700 1 --2- 172.21.15.7:0/3850098619 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f04b0101970 0x7f04b0198ca0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f04a400ec10 tx=0x7f04a400ef20 comp rx=0 tx=0).stop 2023-07-20T02:51:25.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.609+0000 7f04b67ff700 1 -- 172.21.15.7:0/3850098619 shutdown_connections 2023-07-20T02:51:25.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.609+0000 7f04b67ff700 1 --2- 172.21.15.7:0/3850098619 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f04980a1220 0x7f04980a36e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:25.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.609+0000 7f04b67ff700 1 --2- 172.21.15.7:0/3850098619 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04b0102310 0x7f04b019fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:25.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.609+0000 7f04b67ff700 1 --2- 172.21.15.7:0/3850098619 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f04b0100780 0x7f04b0198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:25.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.609+0000 7f04b67ff700 1 --2- 172.21.15.7:0/3850098619 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f04b0101970 0x7f04b0198ca0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:25.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.609+0000 7f04b67ff700 1 -- 172.21.15.7:0/3850098619 >> 172.21.15.7:0/3850098619 conn(0x7f04b00fbcb0 msgr2=0x7f04b00fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:25.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.609+0000 7f04b67ff700 1 -- 172.21.15.7:0/3850098619 shutdown_connections 2023-07-20T02:51:25.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.609+0000 7f04b67ff700 1 -- 172.21.15.7:0/3850098619 wait complete. 2023-07-20T02:51:25.641 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-07-20T02:51:25.642 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-07-20T02:51:25.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.855+0000 7f3423bff700 1 -- 172.21.15.7:0/3505951352 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f341c1019c0 msgr2=0x7f341c101e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:25.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.855+0000 7f3423bff700 1 --2- 172.21.15.7:0/3505951352 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f341c1019c0 0x7f341c101e20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f340c0099c0 tx=0x7f340c009cd0 comp rx=0 tx=0).stop 2023-07-20T02:51:25.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.855+0000 7f3423bff700 1 -- 172.21.15.7:0/3505951352 shutdown_connections 2023-07-20T02:51:25.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.855+0000 7f3423bff700 1 --2- 172.21.15.7:0/3505951352 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f341c102360 0x7f341c108c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:25.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.855+0000 7f3423bff700 1 --2- 172.21.15.7:0/3505951352 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f341c1007d0 0x7f341c100bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:25.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.855+0000 7f3423bff700 1 --2- 172.21.15.7:0/3505951352 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f341c1019c0 0x7f341c101e20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:25.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.855+0000 7f3423bff700 1 -- 172.21.15.7:0/3505951352 >> 172.21.15.7:0/3505951352 conn(0x7f341c0fbd40 msgr2=0x7f341c0fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:25.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.855+0000 7f3423bff700 1 -- 172.21.15.7:0/3505951352 shutdown_connections 2023-07-20T02:51:25.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.855+0000 7f3423bff700 1 -- 172.21.15.7:0/3505951352 wait complete. 2023-07-20T02:51:25.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.856+0000 7f3423bff700 1 Processor -- start 2023-07-20T02:51:25.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.856+0000 7f3423bff700 1 -- start start 2023-07-20T02:51:25.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.856+0000 7f3423bff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f341c1007d0 0x7f341c198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:25.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.856+0000 7f3423bff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f341c1019c0 0x7f341c198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:25.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.856+0000 7f3423bff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f341c102360 0x7f341c19fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:25.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.856+0000 7f3423bff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341c1a0190 con 0x7f341c1019c0 2023-07-20T02:51:25.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.856+0000 7f3423bff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341c1a02d0 con 0x7f341c1007d0 2023-07-20T02:51:25.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.856+0000 7f3423bff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341c1a0460 con 0x7f341c102360 2023-07-20T02:51:25.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.857+0000 7f342199b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f341c1007d0 0x7f341c198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:25.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.857+0000 7f342119a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f341c1019c0 0x7f341c198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:25.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.857+0000 7f342199b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f341c1007d0 0x7f341c198610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.7:55862/0 (socket says 172.21.15.7:55862) 2023-07-20T02:51:25.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.857+0000 7f342199b700 1 -- 172.21.15.7:0/1275589232 learned_addr learned my addr 172.21.15.7:0/1275589232 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:25.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.857+0000 7f342219c700 1 --2- 172.21.15.7:0/1275589232 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f341c102360 0x7f341c19fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:25.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.857+0000 7f342199b700 1 -- 172.21.15.7:0/1275589232 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f341c102360 msgr2=0x7f341c19fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:25.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.857+0000 7f342199b700 1 --2- 172.21.15.7:0/1275589232 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f341c102360 0x7f341c19fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:25.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.857+0000 7f342199b700 1 -- 172.21.15.7:0/1275589232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f341c1019c0 msgr2=0x7f341c198b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:25.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.857+0000 7f342199b700 1 --2- 172.21.15.7:0/1275589232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f341c1019c0 0x7f341c198b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:25.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.857+0000 7f342199b700 1 -- 172.21.15.7:0/1275589232 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f340c0095f0 con 0x7f341c1007d0 2023-07-20T02:51:25.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.857+0000 7f342219c700 1 --2- 172.21.15.7:0/1275589232 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f341c102360 0x7f341c19fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T02:51:25.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.857+0000 7f342199b700 1 --2- 172.21.15.7:0/1275589232 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f341c1007d0 0x7f341c198610 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f341800e970 tx=0x7f341800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:25.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.857+0000 7f3412ffd700 1 -- 172.21.15.7:0/1275589232 <== mon.2 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3418009790 con 0x7f341c1007d0 2023-07-20T02:51:25.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.858+0000 7f3423bff700 1 -- 172.21.15.7:0/1275589232 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f341c1a0740 con 0x7f341c1007d0 2023-07-20T02:51:25.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.858+0000 7f3412ffd700 1 -- 172.21.15.7:0/1275589232 <== mon.2 v2:172.21.15.158:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f341800eec0 con 0x7f341c1007d0 2023-07-20T02:51:25.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.858+0000 7f3412ffd700 1 -- 172.21.15.7:0/1275589232 <== mon.2 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3418003cf0 con 0x7f341c1007d0 2023-07-20T02:51:25.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.858+0000 7f3423bff700 1 -- 172.21.15.7:0/1275589232 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f341c1a0ca0 con 0x7f341c1007d0 2023-07-20T02:51:25.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.859+0000 7f3423bff700 1 -- 172.21.15.7:0/1275589232 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f341c192d70 con 0x7f341c1007d0 2023-07-20T02:51:25.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.861+0000 7f3412ffd700 1 -- 172.21.15.7:0/1275589232 <== mon.2 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f341800ccd0 con 0x7f341c1007d0 2023-07-20T02:51:25.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.862+0000 7f3412ffd700 1 --2- 172.21.15.7:0/1275589232 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f34080a11d0 0x7f34080a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:25.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.862+0000 7f3412ffd700 1 -- 172.21.15.7:0/1275589232 <== mon.2 v2:172.21.15.158:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3418014070 con 0x7f341c1007d0 2023-07-20T02:51:25.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.863+0000 7f342119a700 1 --2- 172.21.15.7:0/1275589232 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f34080a11d0 0x7f34080a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:25.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.863+0000 7f342119a700 1 --2- 172.21.15.7:0/1275589232 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f34080a11d0 0x7f34080a3690 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f340c00b580 tx=0x7f340c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:25.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:25.865+0000 7f3412ffd700 1 -- 172.21.15.7:0/1275589232 <== mon.2 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3418096800 con 0x7f341c1007d0 2023-07-20T02:51:26.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:26.106+0000 7f3423bff700 1 -- 172.21.15.7:0/1275589232 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f341c02c970 con 0x7f341c1007d0 2023-07-20T02:51:26.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:26 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/116505691' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T02:51:26.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:26 smithi158 ceph-mon[128925]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:26.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:26 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/3850098619' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-20T02:51:26.392 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:26 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:26.392 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:26 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:26.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:26 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/116505691' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T02:51:26.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:26 smithi007 ceph-mon[126353]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:26.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:26 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/3850098619' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-20T02:51:26.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:26 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:26.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:26 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:26.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:26 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/116505691' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T02:51:26.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:26 smithi188 ceph-mon[129553]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:26.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:26 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/3850098619' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-20T02:51:26.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:26 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:26.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:26 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:27.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.088+0000 7f3412ffd700 1 -- 172.21.15.7:0/1275589232 <== mon.2 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v80) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3418099e20 con 0x7f341c1007d0 2023-07-20T02:51:27.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.093+0000 7f3423bff700 1 -- 172.21.15.7:0/1275589232 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f34080a11d0 msgr2=0x7f34080a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:27.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.093+0000 7f3423bff700 1 --2- 172.21.15.7:0/1275589232 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f34080a11d0 0x7f34080a3690 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f340c00b580 tx=0x7f340c005fb0 comp rx=0 tx=0).stop 2023-07-20T02:51:27.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.093+0000 7f3423bff700 1 -- 172.21.15.7:0/1275589232 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f341c1007d0 msgr2=0x7f341c198610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:27.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.093+0000 7f3423bff700 1 --2- 172.21.15.7:0/1275589232 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f341c1007d0 0x7f341c198610 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f341800e970 tx=0x7f341800ec80 comp rx=0 tx=0).stop 2023-07-20T02:51:27.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.094+0000 7f3423bff700 1 -- 172.21.15.7:0/1275589232 shutdown_connections 2023-07-20T02:51:27.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.094+0000 7f3423bff700 1 --2- 172.21.15.7:0/1275589232 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f341c102360 0x7f341c19fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:27.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.094+0000 7f3423bff700 1 --2- 172.21.15.7:0/1275589232 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f341c1007d0 0x7f341c198610 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:27.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.094+0000 7f3423bff700 1 --2- 172.21.15.7:0/1275589232 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f34080a11d0 0x7f34080a3690 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:27.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.094+0000 7f3423bff700 1 --2- 172.21.15.7:0/1275589232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f341c1019c0 0x7f341c198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:27.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.094+0000 7f3423bff700 1 -- 172.21.15.7:0/1275589232 >> 172.21.15.7:0/1275589232 conn(0x7f341c0fbd40 msgr2=0x7f341c1072c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:27.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.094+0000 7f3423bff700 1 -- 172.21.15.7:0/1275589232 shutdown_connections 2023-07-20T02:51:27.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.094+0000 7f3423bff700 1 -- 172.21.15.7:0/1275589232 wait complete. 2023-07-20T02:51:27.108 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'cephfs_metadata' created 2023-07-20T02:51:27.127 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2023-07-20T02:51:27.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.344+0000 7ff52a9ff700 1 -- 172.21.15.7:0/3817538519 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff524074770 msgr2=0x7ff524072bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:27.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.344+0000 7ff52a9ff700 1 --2- 172.21.15.7:0/3817538519 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff524074770 0x7ff524072bc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff514009990 tx=0x7ff514009ca0 comp rx=0 tx=0).stop 2023-07-20T02:51:27.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.344+0000 7ff52a9ff700 1 -- 172.21.15.7:0/3817538519 shutdown_connections 2023-07-20T02:51:27.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.344+0000 7ff52a9ff700 1 --2- 172.21.15.7:0/3817538519 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff524108610 0x7ff52410aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:27.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.344+0000 7ff52a9ff700 1 --2- 172.21.15.7:0/3817538519 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff524074770 0x7ff524072bc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:27.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.344+0000 7ff52a9ff700 1 --2- 172.21.15.7:0/3817538519 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff524073190 0x7ff5240735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:27.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.344+0000 7ff52a9ff700 1 -- 172.21.15.7:0/3817538519 >> 172.21.15.7:0/3817538519 conn(0x7ff5240fbcb0 msgr2=0x7ff5240fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:27.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.344+0000 7ff52a9ff700 1 -- 172.21.15.7:0/3817538519 shutdown_connections 2023-07-20T02:51:27.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.344+0000 7ff52a9ff700 1 -- 172.21.15.7:0/3817538519 wait complete. 2023-07-20T02:51:27.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.345+0000 7ff52a9ff700 1 Processor -- start 2023-07-20T02:51:27.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.345+0000 7ff52a9ff700 1 -- start start 2023-07-20T02:51:27.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.346+0000 7ff52a9ff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff524073190 0x7ff524198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:27.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.346+0000 7ff52a9ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff524074770 0x7ff524198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:27.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.346+0000 7ff52a9ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff524108610 0x7ff52419fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:27.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.346+0000 7ff52a9ff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5241a01e0 con 0x7ff524074770 2023-07-20T02:51:27.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.346+0000 7ff52a9ff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5241a0320 con 0x7ff524073190 2023-07-20T02:51:27.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.346+0000 7ff52a9ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5241a04b0 con 0x7ff524108610 2023-07-20T02:51:27.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.346+0000 7ff523fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff524073190 0x7ff524198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:27.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.346+0000 7ff523fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff524073190 0x7ff524198660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.7:55878/0 (socket says 172.21.15.7:55878) 2023-07-20T02:51:27.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.346+0000 7ff523fff700 1 -- 172.21.15.7:0/2620007321 learned_addr learned my addr 172.21.15.7:0/2620007321 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:27.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.346+0000 7ff5237fe700 1 --2- 172.21.15.7:0/2620007321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff524074770 0x7ff524198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:27.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.346+0000 7ff528f9c700 1 --2- 172.21.15.7:0/2620007321 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff524108610 0x7ff52419fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:27.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.346+0000 7ff523fff700 1 -- 172.21.15.7:0/2620007321 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff524108610 msgr2=0x7ff52419fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:27.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.346+0000 7ff523fff700 1 --2- 172.21.15.7:0/2620007321 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff524108610 0x7ff52419fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:27.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.346+0000 7ff523fff700 1 -- 172.21.15.7:0/2620007321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff524074770 msgr2=0x7ff524198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:27.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.346+0000 7ff523fff700 1 --2- 172.21.15.7:0/2620007321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff524074770 0x7ff524198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:27.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.346+0000 7ff523fff700 1 -- 172.21.15.7:0/2620007321 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5140095f0 con 0x7ff524073190 2023-07-20T02:51:27.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.347+0000 7ff523fff700 1 --2- 172.21.15.7:0/2620007321 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff524073190 0x7ff524198660 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff514000c00 tx=0x7ff5140057a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:27.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.347+0000 7ff5217fa700 1 -- 172.21.15.7:0/2620007321 <== mon.2 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51401d070 con 0x7ff524073190 2023-07-20T02:51:27.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.347+0000 7ff52a9ff700 1 -- 172.21.15.7:0/2620007321 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5241a0730 con 0x7ff524073190 2023-07-20T02:51:27.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.347+0000 7ff5217fa700 1 -- 172.21.15.7:0/2620007321 <== mon.2 v2:172.21.15.158:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff51400fb90 con 0x7ff524073190 2023-07-20T02:51:27.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.347+0000 7ff5217fa700 1 -- 172.21.15.7:0/2620007321 <== mon.2 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5140173f0 con 0x7ff524073190 2023-07-20T02:51:27.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.347+0000 7ff52a9ff700 1 -- 172.21.15.7:0/2620007321 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5241a0be0 con 0x7ff524073190 2023-07-20T02:51:27.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.348+0000 7ff52a9ff700 1 -- 172.21.15.7:0/2620007321 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff524106d10 con 0x7ff524073190 2023-07-20T02:51:27.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.351+0000 7ff5217fa700 1 -- 172.21.15.7:0/2620007321 <== mon.2 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7ff514021410 con 0x7ff524073190 2023-07-20T02:51:27.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.352+0000 7ff5217fa700 1 --2- 172.21.15.7:0/2620007321 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7ff50c0a1150 0x7ff50c0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:27.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.352+0000 7ff5217fa700 1 -- 172.21.15.7:0/2620007321 <== mon.2 v2:172.21.15.158:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7497+0+0 (secure 0 0 0) 0x7ff5140cdb40 con 0x7ff524073190 2023-07-20T02:51:27.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.352+0000 7ff5237fe700 1 --2- 172.21.15.7:0/2620007321 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7ff50c0a1150 0x7ff50c0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:27.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.353+0000 7ff5237fe700 1 --2- 172.21.15.7:0/2620007321 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7ff50c0a1150 0x7ff50c0a3610 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff518009c60 tx=0x7ff518009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:27.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.354+0000 7ff5217fa700 1 -- 172.21.15.7:0/2620007321 <== mon.2 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff5140975b0 con 0x7ff524073190 2023-07-20T02:51:27.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:27 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:27.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:27 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:27.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:27 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/1275589232' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-20T02:51:27.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:27 smithi158 ceph-mon[128925]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-20T02:51:27.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:27 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:27.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:27 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:27.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:27 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:51:27.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:27 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:51:27.392 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:27 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:27.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:27 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:27.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:27 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:27.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:27 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/1275589232' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-20T02:51:27.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:27 smithi007 ceph-mon[126353]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-20T02:51:27.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:27 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:27.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:27 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:27.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:27 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:51:27.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:27 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:51:27.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:27 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:27.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:27 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:27.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:27 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:27.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:27 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/1275589232' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-20T02:51:27.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:27 smithi188 ceph-mon[129553]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-20T02:51:27.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:27 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:27.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:27 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:27.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:27 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:51:27.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:27 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:51:27.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:27 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:27.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:27.566+0000 7ff52a9ff700 1 -- 172.21.15.7:0/2620007321 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7ff52402c970 con 0x7ff524073190 2023-07-20T02:51:28.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.096+0000 7ff5217fa700 1 -- 172.21.15.7:0/2620007321 <== mon.2 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v81) v1 ==== 171+0+0 (secure 0 0 0) 0x7ff51409abd0 con 0x7ff524073190 2023-07-20T02:51:28.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.106+0000 7ff52a9ff700 1 -- 172.21.15.7:0/2620007321 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7ff50c0a1150 msgr2=0x7ff50c0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:28.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.106+0000 7ff52a9ff700 1 --2- 172.21.15.7:0/2620007321 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7ff50c0a1150 0x7ff50c0a3610 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff518009c60 tx=0x7ff518009260 comp rx=0 tx=0).stop 2023-07-20T02:51:28.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.106+0000 7ff52a9ff700 1 -- 172.21.15.7:0/2620007321 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff524073190 msgr2=0x7ff524198660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:28.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.106+0000 7ff52a9ff700 1 --2- 172.21.15.7:0/2620007321 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff524073190 0x7ff524198660 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff514000c00 tx=0x7ff5140057a0 comp rx=0 tx=0).stop 2023-07-20T02:51:28.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.107+0000 7ff52a9ff700 1 -- 172.21.15.7:0/2620007321 shutdown_connections 2023-07-20T02:51:28.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.107+0000 7ff52a9ff700 1 --2- 172.21.15.7:0/2620007321 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff524108610 0x7ff52419fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:28.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.107+0000 7ff52a9ff700 1 --2- 172.21.15.7:0/2620007321 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff524073190 0x7ff524198660 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff514000c00 tx=0x7ff5140057a0 comp rx=0 tx=0).stop 2023-07-20T02:51:28.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.107+0000 7ff52a9ff700 1 --2- 172.21.15.7:0/2620007321 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7ff50c0a1150 0x7ff50c0a3610 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:28.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.107+0000 7ff52a9ff700 1 --2- 172.21.15.7:0/2620007321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff524074770 0x7ff524198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:28.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.107+0000 7ff52a9ff700 1 -- 172.21.15.7:0/2620007321 >> 172.21.15.7:0/2620007321 conn(0x7ff5240fbcb0 msgr2=0x7ff5240fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:28.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.108+0000 7ff52a9ff700 1 -- 172.21.15.7:0/2620007321 shutdown_connections 2023-07-20T02:51:28.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.111+0000 7ff52a9ff700 1 -- 172.21.15.7:0/2620007321 wait complete. 2023-07-20T02:51:28.126 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'cephfs_data' created 2023-07-20T02:51:28.146 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-07-20T02:51:28.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.351+0000 7fa6467ff700 1 -- 172.21.15.7:0/1274360061 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa640100760 msgr2=0x7fa640100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:28.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.351+0000 7fa6467ff700 1 --2- 172.21.15.7:0/1274360061 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa640100760 0x7fa640100b80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa630009990 tx=0x7fa630009ca0 comp rx=0 tx=0).stop 2023-07-20T02:51:28.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.351+0000 7fa6467ff700 1 -- 172.21.15.7:0/1274360061 shutdown_connections 2023-07-20T02:51:28.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.351+0000 7fa6467ff700 1 --2- 172.21.15.7:0/1274360061 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6401022f0 0x7fa640108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:28.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.351+0000 7fa6467ff700 1 --2- 172.21.15.7:0/1274360061 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa640100760 0x7fa640100b80 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:28.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.351+0000 7fa6467ff700 1 --2- 172.21.15.7:0/1274360061 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa640101950 0x7fa640101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:28.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.351+0000 7fa6467ff700 1 -- 172.21.15.7:0/1274360061 >> 172.21.15.7:0/1274360061 conn(0x7fa6400fbcb0 msgr2=0x7fa6400fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:28.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.351+0000 7fa6467ff700 1 -- 172.21.15.7:0/1274360061 shutdown_connections 2023-07-20T02:51:28.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.351+0000 7fa6467ff700 1 -- 172.21.15.7:0/1274360061 wait complete. 2023-07-20T02:51:28.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.353+0000 7fa6467ff700 1 Processor -- start 2023-07-20T02:51:28.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.353+0000 7fa6467ff700 1 -- start start 2023-07-20T02:51:28.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.353+0000 7fa6467ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa640100760 0x7fa6401986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:28.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.353+0000 7fa6467ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa640101950 0x7fa640198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:28.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.353+0000 7fa6467ff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6401022f0 0x7fa64019fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:28.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.353+0000 7fa6467ff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6401a0260 con 0x7fa640101950 2023-07-20T02:51:28.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.353+0000 7fa6467ff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6401a03a0 con 0x7fa6401022f0 2023-07-20T02:51:28.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.353+0000 7fa6467ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6401a0530 con 0x7fa640100760 2023-07-20T02:51:28.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.354+0000 7fa63ffff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa640100760 0x7fa6401986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:28.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.354+0000 7fa63ffff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa640100760 0x7fa6401986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.7:54028/0 (socket says 172.21.15.7:54028) 2023-07-20T02:51:28.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.354+0000 7fa63ffff700 1 -- 172.21.15.7:0/2566564960 learned_addr learned my addr 172.21.15.7:0/2566564960 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:28.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.354+0000 7fa644d9c700 1 --2- 172.21.15.7:0/2566564960 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6401022f0 0x7fa64019fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:28.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.354+0000 7fa63f7fe700 1 --2- 172.21.15.7:0/2566564960 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa640101950 0x7fa640198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:28.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.354+0000 7fa63ffff700 1 -- 172.21.15.7:0/2566564960 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6401022f0 msgr2=0x7fa64019fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:28.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.354+0000 7fa63ffff700 1 --2- 172.21.15.7:0/2566564960 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6401022f0 0x7fa64019fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:28.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.354+0000 7fa63ffff700 1 -- 172.21.15.7:0/2566564960 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa640101950 msgr2=0x7fa640198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:28.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.354+0000 7fa63ffff700 1 --2- 172.21.15.7:0/2566564960 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa640101950 0x7fa640198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:28.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.354+0000 7fa63ffff700 1 -- 172.21.15.7:0/2566564960 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6300095f0 con 0x7fa640100760 2023-07-20T02:51:28.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.354+0000 7fa63ffff700 1 --2- 172.21.15.7:0/2566564960 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa640100760 0x7fa6401986e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa630005720 tx=0x7fa630009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:28.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.355+0000 7fa63d7fa700 1 -- 172.21.15.7:0/2566564960 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63001d070 con 0x7fa640100760 2023-07-20T02:51:28.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.355+0000 7fa6467ff700 1 -- 172.21.15.7:0/2566564960 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6401a07b0 con 0x7fa640100760 2023-07-20T02:51:28.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.355+0000 7fa63d7fa700 1 -- 172.21.15.7:0/2566564960 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa630021410 con 0x7fa640100760 2023-07-20T02:51:28.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.355+0000 7fa63d7fa700 1 -- 172.21.15.7:0/2566564960 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa630017650 con 0x7fa640100760 2023-07-20T02:51:28.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.355+0000 7fa6467ff700 1 -- 172.21.15.7:0/2566564960 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6401a0be0 con 0x7fa640100760 2023-07-20T02:51:28.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.356+0000 7fa6467ff700 1 -- 172.21.15.7:0/2566564960 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa64010bfd0 con 0x7fa640100760 2023-07-20T02:51:28.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.358+0000 7fa63d7fa700 1 -- 172.21.15.7:0/2566564960 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fa630021580 con 0x7fa640100760 2023-07-20T02:51:28.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.359+0000 7fa63d7fa700 1 --2- 172.21.15.7:0/2566564960 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fa6280a1150 0x7fa6280a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:28.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.359+0000 7fa63d7fa700 1 -- 172.21.15.7:0/2566564960 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7879+0+0 (secure 0 0 0) 0x7fa6300cd940 con 0x7fa640100760 2023-07-20T02:51:28.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.359+0000 7fa63f7fe700 1 --2- 172.21.15.7:0/2566564960 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fa6280a1150 0x7fa6280a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:28.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.360+0000 7fa63f7fe700 1 --2- 172.21.15.7:0/2566564960 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fa6280a1150 0x7fa6280a3610 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa634009c60 tx=0x7fa634009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:28.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.365+0000 7fa63d7fa700 1 -- 172.21.15.7:0/2566564960 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa630097230 con 0x7fa640100760 2023-07-20T02:51:28.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:28 smithi158 ceph-mon[128925]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-20T02:51:28.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:28 smithi158 ceph-mon[128925]: osdmap e80: 12 total, 12 up, 12 in 2023-07-20T02:51:28.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:28 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:51:28.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:28 smithi158 ceph-mon[128925]: pgmap v267: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:28.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:28 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/2620007321' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-07-20T02:51:28.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:28 smithi158 ceph-mon[128925]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-07-20T02:51:28.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:28 smithi007 ceph-mon[126353]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-20T02:51:28.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:28 smithi007 ceph-mon[126353]: osdmap e80: 12 total, 12 up, 12 in 2023-07-20T02:51:28.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:28 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:51:28.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:28 smithi007 ceph-mon[126353]: pgmap v267: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:28.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:28 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/2620007321' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-07-20T02:51:28.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:28 smithi007 ceph-mon[126353]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-07-20T02:51:28.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:28 smithi188 ceph-mon[129553]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-20T02:51:28.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:28 smithi188 ceph-mon[129553]: osdmap e80: 12 total, 12 up, 12 in 2023-07-20T02:51:28.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:28 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T02:51:28.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:28 smithi188 ceph-mon[129553]: pgmap v267: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:28.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:28 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/2620007321' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-07-20T02:51:28.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:28 smithi188 ceph-mon[129553]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-07-20T02:51:28.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:28.638+0000 7fa6467ff700 1 -- 172.21.15.7:0/2566564960 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fa64010c160 con 0x7fa640100760 2023-07-20T02:51:29.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.147+0000 7fa63d7fa700 1 -- 172.21.15.7:0/2566564960 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7fa6300260c0 con 0x7fa640100760 2023-07-20T02:51:29.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.151+0000 7fa6467ff700 1 -- 172.21.15.7:0/2566564960 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fa6280a1150 msgr2=0x7fa6280a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:29.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.151+0000 7fa6467ff700 1 --2- 172.21.15.7:0/2566564960 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fa6280a1150 0x7fa6280a3610 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa634009c60 tx=0x7fa634009260 comp rx=0 tx=0).stop 2023-07-20T02:51:29.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.151+0000 7fa6467ff700 1 -- 172.21.15.7:0/2566564960 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa640100760 msgr2=0x7fa6401986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:29.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.151+0000 7fa6467ff700 1 --2- 172.21.15.7:0/2566564960 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa640100760 0x7fa6401986e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa630005720 tx=0x7fa630009580 comp rx=0 tx=0).stop 2023-07-20T02:51:29.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.152+0000 7fa6467ff700 1 -- 172.21.15.7:0/2566564960 shutdown_connections 2023-07-20T02:51:29.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.152+0000 7fa6467ff700 1 --2- 172.21.15.7:0/2566564960 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa6401022f0 0x7fa64019fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:29.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.152+0000 7fa6467ff700 1 --2- 172.21.15.7:0/2566564960 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa640100760 0x7fa6401986e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:29.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.152+0000 7fa6467ff700 1 --2- 172.21.15.7:0/2566564960 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fa6280a1150 0x7fa6280a3610 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:29.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.152+0000 7fa6467ff700 1 --2- 172.21.15.7:0/2566564960 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa640101950 0x7fa640198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:29.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.152+0000 7fa6467ff700 1 -- 172.21.15.7:0/2566564960 >> 172.21.15.7:0/2566564960 conn(0x7fa6400fbcb0 msgr2=0x7fa6400fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:29.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.152+0000 7fa6467ff700 1 -- 172.21.15.7:0/2566564960 shutdown_connections 2023-07-20T02:51:29.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.152+0000 7fa6467ff700 1 -- 172.21.15.7:0/2566564960 wait complete. 2023-07-20T02:51:29.166 INFO:teuthology.orchestra.run.smithi007.stderr:new fs with metadata pool 2 and data pool 3 2023-07-20T02:51:29.185 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2023-07-20T02:51:29.186 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2023-07-20T02:51:29.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:29 smithi158 ceph-mon[128925]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-07-20T02:51:29.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:29 smithi158 ceph-mon[128925]: osdmap e81: 12 total, 12 up, 12 in 2023-07-20T02:51:29.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:29 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/2566564960' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-20T02:51:29.392 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:29 smithi158 ceph-mon[128925]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-20T02:51:29.392 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:29 smithi158 ceph-mon[128925]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-20T02:51:29.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.398+0000 7f74461ff700 1 -- 172.21.15.7:0/2728200856 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7440100780 msgr2=0x7f7440100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:29.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.398+0000 7f74461ff700 1 --2- 172.21.15.7:0/2728200856 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7440100780 0x7f7440100ba0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f74300099c0 tx=0x7f7430009cd0 comp rx=0 tx=0).stop 2023-07-20T02:51:29.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.398+0000 7f74461ff700 1 -- 172.21.15.7:0/2728200856 shutdown_connections 2023-07-20T02:51:29.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.398+0000 7f74461ff700 1 --2- 172.21.15.7:0/2728200856 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7440102310 0x7f7440108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:29.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.398+0000 7f74461ff700 1 --2- 172.21.15.7:0/2728200856 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7440100780 0x7f7440100ba0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:29.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.398+0000 7f74461ff700 1 --2- 172.21.15.7:0/2728200856 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7440101970 0x7f7440101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:29.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.398+0000 7f74461ff700 1 -- 172.21.15.7:0/2728200856 >> 172.21.15.7:0/2728200856 conn(0x7f74400fbcf0 msgr2=0x7f74400fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:29.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.399+0000 7f74461ff700 1 -- 172.21.15.7:0/2728200856 shutdown_connections 2023-07-20T02:51:29.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.399+0000 7f74461ff700 1 -- 172.21.15.7:0/2728200856 wait complete. 2023-07-20T02:51:29.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.400+0000 7f74461ff700 1 Processor -- start 2023-07-20T02:51:29.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.400+0000 7f74461ff700 1 -- start start 2023-07-20T02:51:29.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.400+0000 7f74461ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7440100780 0x7f7440194270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:29.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.400+0000 7f74461ff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7440101970 0x7f74401947b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:29.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.400+0000 7f74461ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7440102310 0x7f744019b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:29.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.400+0000 7f74461ff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744019bdf0 con 0x7f7440102310 2023-07-20T02:51:29.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.400+0000 7f74461ff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744019bf30 con 0x7f7440101970 2023-07-20T02:51:29.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.400+0000 7f74461ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744019c0c0 con 0x7f7440100780 2023-07-20T02:51:29.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.400+0000 7f743f7fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7440100780 0x7f7440194270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:29.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.400+0000 7f743f7fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7440100780 0x7f7440194270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.7:54050/0 (socket says 172.21.15.7:54050) 2023-07-20T02:51:29.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.400+0000 7f743f7fe700 1 -- 172.21.15.7:0/1071566232 learned_addr learned my addr 172.21.15.7:0/1071566232 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:29.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.401+0000 7f743effd700 1 --2- 172.21.15.7:0/1071566232 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7440101970 0x7f74401947b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:29.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.401+0000 7f743ffff700 1 --2- 172.21.15.7:0/1071566232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7440102310 0x7f744019b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:29.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.401+0000 7f743f7fe700 1 -- 172.21.15.7:0/1071566232 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7440101970 msgr2=0x7f74401947b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:29.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.401+0000 7f743f7fe700 1 --2- 172.21.15.7:0/1071566232 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7440101970 0x7f74401947b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:29.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.401+0000 7f743f7fe700 1 -- 172.21.15.7:0/1071566232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7440102310 msgr2=0x7f744019b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:29.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.401+0000 7f743f7fe700 1 --2- 172.21.15.7:0/1071566232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7440102310 0x7f744019b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:29.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.401+0000 7f743f7fe700 1 -- 172.21.15.7:0/1071566232 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74300095f0 con 0x7f7440100780 2023-07-20T02:51:29.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.401+0000 7f743ffff700 1 --2- 172.21.15.7:0/1071566232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7440102310 0x7f744019b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T02:51:29.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.401+0000 7f743f7fe700 1 --2- 172.21.15.7:0/1071566232 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7440100780 0x7f7440194270 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7430009990 tx=0x7f743000f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:29.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.401+0000 7f743cff9700 1 -- 172.21.15.7:0/1071566232 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743001d070 con 0x7f7440100780 2023-07-20T02:51:29.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.401+0000 7f743cff9700 1 -- 172.21.15.7:0/1071566232 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f74300054f0 con 0x7f7440100780 2023-07-20T02:51:29.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.401+0000 7f743cff9700 1 -- 172.21.15.7:0/1071566232 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7430004b00 con 0x7f7440100780 2023-07-20T02:51:29.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.401+0000 7f74461ff700 1 -- 172.21.15.7:0/1071566232 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f744019c340 con 0x7f7440100780 2023-07-20T02:51:29.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.402+0000 7f74461ff700 1 -- 172.21.15.7:0/1071566232 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f744019c710 con 0x7f7440100780 2023-07-20T02:51:29.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.402+0000 7f74461ff700 1 -- 172.21.15.7:0/1071566232 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7440107d40 con 0x7f7440100780 2023-07-20T02:51:29.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.406+0000 7f743cff9700 1 -- 172.21.15.7:0/1071566232 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f743000fb80 con 0x7f7440100780 2023-07-20T02:51:29.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.408+0000 7f743cff9700 1 --2- 172.21.15.7:0/1071566232 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f74280a11d0 0x7f74280a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:29.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.408+0000 7f743cff9700 1 -- 172.21.15.7:0/1071566232 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f74300cd610 con 0x7f7440100780 2023-07-20T02:51:29.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.408+0000 7f743effd700 1 --2- 172.21.15.7:0/1071566232 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f74280a11d0 0x7f74280a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:29.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.408+0000 7f743cff9700 1 -- 172.21.15.7:0/1071566232 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f743009a4c0 con 0x7f7440100780 2023-07-20T02:51:29.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.409+0000 7f743effd700 1 --2- 172.21.15.7:0/1071566232 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f74280a11d0 0x7f74280a3690 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f743400ff50 tx=0x7f7434009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:29.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:29 smithi007 ceph-mon[126353]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-07-20T02:51:29.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:29 smithi007 ceph-mon[126353]: osdmap e81: 12 total, 12 up, 12 in 2023-07-20T02:51:29.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:29 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/2566564960' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-20T02:51:29.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:29 smithi007 ceph-mon[126353]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-20T02:51:29.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:29 smithi007 ceph-mon[126353]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-20T02:51:29.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:29 smithi188 ceph-mon[129553]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-07-20T02:51:29.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:29 smithi188 ceph-mon[129553]: osdmap e81: 12 total, 12 up, 12 in 2023-07-20T02:51:29.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:29 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/2566564960' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-20T02:51:29.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:29 smithi188 ceph-mon[129553]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-20T02:51:29.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:29 smithi188 ceph-mon[129553]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-20T02:51:29.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:29.642+0000 7f74461ff700 1 -- 172.21.15.7:0/1071566232 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f744002cbe0 con 0x7f7440100780 2023-07-20T02:51:30.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.120+0000 7f743cff9700 1 -- 172.21.15.7:0/1071566232 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v83) v1 ==== 157+0+0 (secure 0 0 0) 0x7f743009a6a0 con 0x7f7440100780 2023-07-20T02:51:30.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.123+0000 7f74461ff700 1 -- 172.21.15.7:0/1071566232 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f74280a11d0 msgr2=0x7f74280a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:30.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.123+0000 7f74461ff700 1 --2- 172.21.15.7:0/1071566232 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f74280a11d0 0x7f74280a3690 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f743400ff50 tx=0x7f7434009260 comp rx=0 tx=0).stop 2023-07-20T02:51:30.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.123+0000 7f74461ff700 1 -- 172.21.15.7:0/1071566232 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7440100780 msgr2=0x7f7440194270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:30.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.123+0000 7f74461ff700 1 --2- 172.21.15.7:0/1071566232 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7440100780 0x7f7440194270 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7430009990 tx=0x7f743000f690 comp rx=0 tx=0).stop 2023-07-20T02:51:30.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.123+0000 7f74461ff700 1 -- 172.21.15.7:0/1071566232 shutdown_connections 2023-07-20T02:51:30.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.123+0000 7f74461ff700 1 --2- 172.21.15.7:0/1071566232 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f74280a11d0 0x7f74280a3690 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:30.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.123+0000 7f74461ff700 1 --2- 172.21.15.7:0/1071566232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7440102310 0x7f744019b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:30.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.123+0000 7f74461ff700 1 --2- 172.21.15.7:0/1071566232 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7440100780 0x7f7440194270 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:30.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.123+0000 7f74461ff700 1 --2- 172.21.15.7:0/1071566232 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7440101970 0x7f74401947b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:30.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.123+0000 7f74461ff700 1 -- 172.21.15.7:0/1071566232 >> 172.21.15.7:0/1071566232 conn(0x7f74400fbcf0 msgr2=0x7f7440104e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:30.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.123+0000 7f74461ff700 1 -- 172.21.15.7:0/1071566232 shutdown_connections 2023-07-20T02:51:30.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.123+0000 7f74461ff700 1 -- 172.21.15.7:0/1071566232 wait complete. 2023-07-20T02:51:30.154 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2023-07-20T02:51:30.390 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:30 smithi158 ceph-mon[128925]: osdmap e82: 12 total, 12 up, 12 in 2023-07-20T02:51:30.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:30 smithi158 ceph-mon[128925]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-20T02:51:30.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:30 smithi158 ceph-mon[128925]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-20T02:51:30.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:30 smithi158 ceph-mon[128925]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-20T02:51:30.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:30 smithi158 ceph-mon[128925]: pgmap v270: 97 pgs: 6 creating+peering, 1 active+clean, 90 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:30.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:30 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/1071566232' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-07-20T02:51:30.392 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:30 smithi158 ceph-mon[128925]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-07-20T02:51:30.392 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:30 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:30.392 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:30 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:30.392 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:30 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:30.392 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:30 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:30.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.393+0000 7f36905ff700 1 -- 172.21.15.7:0/1709919438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3688100780 msgr2=0x7f3688100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:30.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.393+0000 7f36905ff700 1 --2- 172.21.15.7:0/1709919438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3688100780 0x7f3688100ba0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3684009990 tx=0x7f3684009ca0 comp rx=0 tx=0).stop 2023-07-20T02:51:30.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.393+0000 7f36905ff700 1 -- 172.21.15.7:0/1709919438 shutdown_connections 2023-07-20T02:51:30.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.393+0000 7f36905ff700 1 --2- 172.21.15.7:0/1709919438 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3688102310 0x7f3688108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:30.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.393+0000 7f36905ff700 1 --2- 172.21.15.7:0/1709919438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3688100780 0x7f3688100ba0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:30.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.393+0000 7f36905ff700 1 --2- 172.21.15.7:0/1709919438 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3688101970 0x7f3688101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:30.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.393+0000 7f36905ff700 1 -- 172.21.15.7:0/1709919438 >> 172.21.15.7:0/1709919438 conn(0x7f36880fbcb0 msgr2=0x7f36880fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:30.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.393+0000 7f36905ff700 1 -- 172.21.15.7:0/1709919438 shutdown_connections 2023-07-20T02:51:30.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.393+0000 7f36905ff700 1 -- 172.21.15.7:0/1709919438 wait complete. 2023-07-20T02:51:30.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.394+0000 7f36905ff700 1 Processor -- start 2023-07-20T02:51:30.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.394+0000 7f36905ff700 1 -- start start 2023-07-20T02:51:30.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.394+0000 7f36905ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3688100780 0x7f3688074520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:30.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.394+0000 7f36905ff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3688101970 0x7f3688072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:30.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.394+0000 7f36905ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3688102310 0x7f36880730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:30.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.394+0000 7f36905ff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3688073840 con 0x7f3688100780 2023-07-20T02:51:30.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.394+0000 7f36905ff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36880739b0 con 0x7f3688101970 2023-07-20T02:51:30.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.394+0000 7f36905ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36881a04b0 con 0x7f3688102310 2023-07-20T02:51:30.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.394+0000 7f368e39b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3688100780 0x7f3688074520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:30.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.394+0000 7f368db9a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3688101970 0x7f3688072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:30.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.394+0000 7f368e39b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3688100780 0x7f3688074520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60700/0 (socket says 172.21.15.7:60700) 2023-07-20T02:51:30.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.394+0000 7f368e39b700 1 -- 172.21.15.7:0/4260662413 learned_addr learned my addr 172.21.15.7:0/4260662413 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:30.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.394+0000 7f368eb9c700 1 --2- 172.21.15.7:0/4260662413 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3688102310 0x7f36880730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:30.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.395+0000 7f368e39b700 1 -- 172.21.15.7:0/4260662413 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3688102310 msgr2=0x7f36880730a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:30.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.395+0000 7f368e39b700 1 --2- 172.21.15.7:0/4260662413 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3688102310 0x7f36880730a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:30.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.395+0000 7f368e39b700 1 -- 172.21.15.7:0/4260662413 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3688101970 msgr2=0x7f3688072b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:30.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.395+0000 7f368e39b700 1 --2- 172.21.15.7:0/4260662413 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3688101970 0x7f3688072b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:30.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.395+0000 7f368e39b700 1 -- 172.21.15.7:0/4260662413 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36840095f0 con 0x7f3688100780 2023-07-20T02:51:30.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.395+0000 7f368e39b700 1 --2- 172.21.15.7:0/4260662413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3688100780 0x7f3688074520 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3684004710 tx=0x7f3684009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:30.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.395+0000 7f367f7fe700 1 -- 172.21.15.7:0/4260662413 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368401d070 con 0x7f3688100780 2023-07-20T02:51:30.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.395+0000 7f36905ff700 1 -- 172.21.15.7:0/4260662413 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36881a0730 con 0x7f3688100780 2023-07-20T02:51:30.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.395+0000 7f367f7fe700 1 -- 172.21.15.7:0/4260662413 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3684021410 con 0x7f3688100780 2023-07-20T02:51:30.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.395+0000 7f367f7fe700 1 -- 172.21.15.7:0/4260662413 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3684017650 con 0x7f3688100780 2023-07-20T02:51:30.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.395+0000 7f36905ff700 1 -- 172.21.15.7:0/4260662413 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36881a0bb0 con 0x7f3688100780 2023-07-20T02:51:30.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.396+0000 7f36905ff700 1 -- 172.21.15.7:0/4260662413 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3688107c70 con 0x7f3688100780 2023-07-20T02:51:30.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.399+0000 7f367f7fe700 1 -- 172.21.15.7:0/4260662413 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f3684021580 con 0x7f3688100780 2023-07-20T02:51:30.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.400+0000 7f367f7fe700 1 --2- 172.21.15.7:0/4260662413 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f36740a11f0 0x7f36740a36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:30.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.400+0000 7f367f7fe700 1 -- 172.21.15.7:0/4260662413 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8205+0+0 (secure 0 0 0) 0x7f36840ce3b0 con 0x7f3688100780 2023-07-20T02:51:30.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.400+0000 7f368db9a700 1 --2- 172.21.15.7:0/4260662413 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f36740a11f0 0x7f36740a36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:30.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.401+0000 7f368db9a700 1 --2- 172.21.15.7:0/4260662413 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f36740a11f0 0x7f36740a36b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3678009c60 tx=0x7f3678009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:30.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.402+0000 7f367f7fe700 1 -- 172.21.15.7:0/4260662413 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3684097b50 con 0x7f3688100780 2023-07-20T02:51:30.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:30 smithi007 ceph-mon[126353]: osdmap e82: 12 total, 12 up, 12 in 2023-07-20T02:51:30.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:30 smithi007 ceph-mon[126353]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-20T02:51:30.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:30 smithi007 ceph-mon[126353]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-20T02:51:30.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:30 smithi007 ceph-mon[126353]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-20T02:51:30.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:30 smithi007 ceph-mon[126353]: pgmap v270: 97 pgs: 6 creating+peering, 1 active+clean, 90 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:30.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:30 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/1071566232' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-07-20T02:51:30.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:30 smithi007 ceph-mon[126353]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-07-20T02:51:30.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:30 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:30.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:30 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:30.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:30 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:30.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:30 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:30.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:30 smithi188 ceph-mon[129553]: osdmap e82: 12 total, 12 up, 12 in 2023-07-20T02:51:30.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:30 smithi188 ceph-mon[129553]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-20T02:51:30.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:30 smithi188 ceph-mon[129553]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-20T02:51:30.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:30 smithi188 ceph-mon[129553]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-20T02:51:30.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:30 smithi188 ceph-mon[129553]: pgmap v270: 97 pgs: 6 creating+peering, 1 active+clean, 90 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T02:51:30.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:30 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/1071566232' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-07-20T02:51:30.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:30 smithi188 ceph-mon[129553]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-07-20T02:51:30.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:30 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:30.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:30 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:30.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:30 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:30.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:30 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:30.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:30.648+0000 7f36905ff700 1 -- 172.21.15.7:0/4260662413 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f368802cc70 con 0x7f3688100780 2023-07-20T02:51:31.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:31 smithi158 ceph-mon[128925]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-07-20T02:51:31.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:31 smithi158 ceph-mon[128925]: osdmap e83: 12 total, 12 up, 12 in 2023-07-20T02:51:31.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:31 smithi158 ceph-mon[128925]: mds.? [v2:172.21.15.158:6838/2552719389,v1:172.21.15.158:6839/2552719389] up:active 2023-07-20T02:51:31.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:31 smithi158 ceph-mon[128925]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T02:51:31.392 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:31 smithi158 ceph-mon[128925]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T02:51:31.392 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:31 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/4260662413' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-20T02:51:31.392 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:31 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:31.392 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:31 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:31.392 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:31 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:51:31.392 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:31 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:51:31.393 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:31 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:31.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:31 smithi007 ceph-mon[126353]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-07-20T02:51:31.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:31 smithi007 ceph-mon[126353]: osdmap e83: 12 total, 12 up, 12 in 2023-07-20T02:51:31.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:31 smithi007 ceph-mon[126353]: mds.? [v2:172.21.15.158:6838/2552719389,v1:172.21.15.158:6839/2552719389] up:active 2023-07-20T02:51:31.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:31 smithi007 ceph-mon[126353]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T02:51:31.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:31 smithi007 ceph-mon[126353]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T02:51:31.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:31 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/4260662413' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-20T02:51:31.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:31 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:31.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:31 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:31.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:31 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:51:31.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:31 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:51:31.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:31 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:31.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:31 smithi188 ceph-mon[129553]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-07-20T02:51:31.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:31 smithi188 ceph-mon[129553]: osdmap e83: 12 total, 12 up, 12 in 2023-07-20T02:51:31.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:31 smithi188 ceph-mon[129553]: mds.? [v2:172.21.15.158:6838/2552719389,v1:172.21.15.158:6839/2552719389] up:active 2023-07-20T02:51:31.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:31 smithi188 ceph-mon[129553]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T02:51:31.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:31 smithi188 ceph-mon[129553]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T02:51:31.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:31 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/4260662413' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-20T02:51:31.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:31 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:31.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:31 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:31.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:31 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T02:51:31.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:31 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T02:51:31.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:31 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' 2023-07-20T02:51:32.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.170+0000 7f367f7fe700 1 -- 172.21.15.7:0/4260662413 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v85) v1 ==== 228+0+0 (secure 0 0 0) 0x7f368409b170 con 0x7f3688100780 2023-07-20T02:51:32.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.177+0000 7f36905ff700 1 -- 172.21.15.7:0/4260662413 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f36740a11f0 msgr2=0x7f36740a36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:32.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.177+0000 7f36905ff700 1 --2- 172.21.15.7:0/4260662413 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f36740a11f0 0x7f36740a36b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3678009c60 tx=0x7f3678009260 comp rx=0 tx=0).stop 2023-07-20T02:51:32.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.177+0000 7f36905ff700 1 -- 172.21.15.7:0/4260662413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3688100780 msgr2=0x7f3688074520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:32.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.177+0000 7f36905ff700 1 --2- 172.21.15.7:0/4260662413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3688100780 0x7f3688074520 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3684004710 tx=0x7f3684009580 comp rx=0 tx=0).stop 2023-07-20T02:51:32.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.177+0000 7f36905ff700 1 -- 172.21.15.7:0/4260662413 shutdown_connections 2023-07-20T02:51:32.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.177+0000 7f36905ff700 1 --2- 172.21.15.7:0/4260662413 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3688102310 0x7f36880730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:32.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.177+0000 7f36905ff700 1 --2- 172.21.15.7:0/4260662413 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f36740a11f0 0x7f36740a36b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:32.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.177+0000 7f36905ff700 1 --2- 172.21.15.7:0/4260662413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3688100780 0x7f3688074520 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:32.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.177+0000 7f36905ff700 1 --2- 172.21.15.7:0/4260662413 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3688101970 0x7f3688072b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:32.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.177+0000 7f36905ff700 1 -- 172.21.15.7:0/4260662413 >> 172.21.15.7:0/4260662413 conn(0x7f36880fbcb0 msgr2=0x7f3688104db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:32.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.177+0000 7f36905ff700 1 -- 172.21.15.7:0/4260662413 shutdown_connections 2023-07-20T02:51:32.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.177+0000 7f36905ff700 1 -- 172.21.15.7:0/4260662413 wait complete. 2023-07-20T02:51:32.193 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'cephfs_data_ec' created 2023-07-20T02:51:32.210 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2023-07-20T02:51:32.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.407+0000 7f78203ff700 1 -- 172.21.15.7:0/2359408106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78181022f0 msgr2=0x7f7818108ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:32.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.407+0000 7f78203ff700 1 --2- 172.21.15.7:0/2359408106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78181022f0 0x7f7818108ba0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f781000b0d0 tx=0x7f781000b3e0 comp rx=0 tx=0).stop 2023-07-20T02:51:32.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.407+0000 7f78203ff700 1 -- 172.21.15.7:0/2359408106 shutdown_connections 2023-07-20T02:51:32.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.407+0000 7f78203ff700 1 --2- 172.21.15.7:0/2359408106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78181022f0 0x7f7818108ba0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:32.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.407+0000 7f78203ff700 1 --2- 172.21.15.7:0/2359408106 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f7818100760 0x7f7818100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:32.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.407+0000 7f78203ff700 1 --2- 172.21.15.7:0/2359408106 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7818101950 0x7f7818101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:32.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.407+0000 7f78203ff700 1 -- 172.21.15.7:0/2359408106 >> 172.21.15.7:0/2359408106 conn(0x7f78180fbcb0 msgr2=0x7f78180fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:32.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.407+0000 7f78203ff700 1 -- 172.21.15.7:0/2359408106 shutdown_connections 2023-07-20T02:51:32.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.408+0000 7f78203ff700 1 -- 172.21.15.7:0/2359408106 wait complete. 2023-07-20T02:51:32.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.408+0000 7f78203ff700 1 Processor -- start 2023-07-20T02:51:32.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.409+0000 7f78203ff700 1 -- start start 2023-07-20T02:51:32.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.409+0000 7f78203ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7818100760 0x7f7818198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:32.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.409+0000 7f78203ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7818101950 0x7f7818198cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:32.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.409+0000 7f78203ff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f78181022f0 0x7f781819fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:32.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.409+0000 7f78203ff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78181a02f0 con 0x7f7818101950 2023-07-20T02:51:32.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.409+0000 7f78203ff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78181a0430 con 0x7f78181022f0 2023-07-20T02:51:32.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.409+0000 7f78203ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78181a05c0 con 0x7f7818100760 2023-07-20T02:51:32.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.409+0000 7f781d99a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7818101950 0x7f7818198cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:32.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.409+0000 7f781e19b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7818100760 0x7f7818198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:32.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.409+0000 7f781d99a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7818101950 0x7f7818198cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51840/0 (socket says 172.21.15.7:51840) 2023-07-20T02:51:32.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.409+0000 7f781d99a700 1 -- 172.21.15.7:0/3366502505 learned_addr learned my addr 172.21.15.7:0/3366502505 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:32.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.409+0000 7f781e99c700 1 --2- 172.21.15.7:0/3366502505 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f78181022f0 0x7f781819fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:32.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.410+0000 7f781e19b700 1 -- 172.21.15.7:0/3366502505 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f78181022f0 msgr2=0x7f781819fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:32.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.410+0000 7f781e19b700 1 --2- 172.21.15.7:0/3366502505 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f78181022f0 0x7f781819fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:32.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.410+0000 7f781e19b700 1 -- 172.21.15.7:0/3366502505 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7818101950 msgr2=0x7f7818198cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:32.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.410+0000 7f781e19b700 1 --2- 172.21.15.7:0/3366502505 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7818101950 0x7f7818198cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:32.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.410+0000 7f781e19b700 1 -- 172.21.15.7:0/3366502505 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7810009d00 con 0x7f7818100760 2023-07-20T02:51:32.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.410+0000 7f781d99a700 1 --2- 172.21.15.7:0/3366502505 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7818101950 0x7f7818198cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T02:51:32.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.410+0000 7f781e19b700 1 --2- 172.21.15.7:0/3366502505 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7818100760 0x7f7818198770 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f781400e9c0 tx=0x7f781400ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:32.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.410+0000 7f780f7fe700 1 -- 172.21.15.7:0/3366502505 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7814004220 con 0x7f7818100760 2023-07-20T02:51:32.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.410+0000 7f78203ff700 1 -- 172.21.15.7:0/3366502505 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78181a08a0 con 0x7f7818100760 2023-07-20T02:51:32.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.410+0000 7f780f7fe700 1 -- 172.21.15.7:0/3366502505 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7814004380 con 0x7f7818100760 2023-07-20T02:51:32.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.410+0000 7f780f7fe700 1 -- 172.21.15.7:0/3366502505 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7814003be0 con 0x7f7818100760 2023-07-20T02:51:32.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.410+0000 7f78203ff700 1 -- 172.21.15.7:0/3366502505 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78181a0db0 con 0x7f7818100760 2023-07-20T02:51:32.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.411+0000 7f78203ff700 1 -- 172.21.15.7:0/3366502505 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f781810bfd0 con 0x7f7818100760 2023-07-20T02:51:32.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.414+0000 7f780f7fe700 1 -- 172.21.15.7:0/3366502505 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f7814003d40 con 0x7f7818100760 2023-07-20T02:51:32.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.415+0000 7f780f7fe700 1 --2- 172.21.15.7:0/3366502505 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f78040a1150 0x7f78040a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:32.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.415+0000 7f780f7fe700 1 -- 172.21.15.7:0/3366502505 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8698+0+0 (secure 0 0 0) 0x7f7814014070 con 0x7f7818100760 2023-07-20T02:51:32.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.415+0000 7f781d99a700 1 --2- 172.21.15.7:0/3366502505 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f78040a1150 0x7f78040a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:32.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.416+0000 7f781d99a700 1 --2- 172.21.15.7:0/3366502505 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f78040a1150 0x7f78040a3610 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7808011550 tx=0x7f7808009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:32.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.417+0000 7f780f7fe700 1 -- 172.21.15.7:0/3366502505 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7814096490 con 0x7f7818100760 2023-07-20T02:51:32.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:32 smithi007 ceph-mon[126353]: pgmap v273: 97 pgs: 69 active+clean, 6 creating+peering, 22 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 6 op/s 2023-07-20T02:51:32.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:32 smithi007 ceph-mon[126353]: osdmap e84: 12 total, 12 up, 12 in 2023-07-20T02:51:32.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:32 smithi188 ceph-mon[129553]: pgmap v273: 97 pgs: 69 active+clean, 6 creating+peering, 22 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 6 op/s 2023-07-20T02:51:32.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:32 smithi188 ceph-mon[129553]: osdmap e84: 12 total, 12 up, 12 in 2023-07-20T02:51:32.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:32.625+0000 7f78203ff700 1 -- 172.21.15.7:0/3366502505 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f781810c160 con 0x7f7818100760 2023-07-20T02:51:32.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:32 smithi158 ceph-mon[128925]: pgmap v273: 97 pgs: 69 active+clean, 6 creating+peering, 22 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 6 op/s 2023-07-20T02:51:32.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:32 smithi158 ceph-mon[128925]: osdmap e84: 12 total, 12 up, 12 in 2023-07-20T02:51:33.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.174+0000 7f780f7fe700 1 -- 172.21.15.7:0/3366502505 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v86) v1 ==== 169+0+0 (secure 0 0 0) 0x7f781401f340 con 0x7f7818100760 2023-07-20T02:51:33.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.191+0000 7f78203ff700 1 -- 172.21.15.7:0/3366502505 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f78040a1150 msgr2=0x7f78040a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:33.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.191+0000 7f78203ff700 1 --2- 172.21.15.7:0/3366502505 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f78040a1150 0x7f78040a3610 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7808011550 tx=0x7f7808009260 comp rx=0 tx=0).stop 2023-07-20T02:51:33.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.191+0000 7f78203ff700 1 -- 172.21.15.7:0/3366502505 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7818100760 msgr2=0x7f7818198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:33.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.191+0000 7f78203ff700 1 --2- 172.21.15.7:0/3366502505 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7818100760 0x7f7818198770 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f781400e9c0 tx=0x7f781400ecd0 comp rx=0 tx=0).stop 2023-07-20T02:51:33.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.195+0000 7f78203ff700 1 -- 172.21.15.7:0/3366502505 shutdown_connections 2023-07-20T02:51:33.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.195+0000 7f78203ff700 1 --2- 172.21.15.7:0/3366502505 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f78181022f0 0x7f781819fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:33.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.195+0000 7f78203ff700 1 --2- 172.21.15.7:0/3366502505 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7818100760 0x7f7818198770 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:33.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.195+0000 7f78203ff700 1 --2- 172.21.15.7:0/3366502505 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f78040a1150 0x7f78040a3610 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:33.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.195+0000 7f78203ff700 1 --2- 172.21.15.7:0/3366502505 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7818101950 0x7f7818198cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:33.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.195+0000 7f78203ff700 1 -- 172.21.15.7:0/3366502505 >> 172.21.15.7:0/3366502505 conn(0x7f78180fbcb0 msgr2=0x7f78181071e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:33.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.197+0000 7f78203ff700 1 -- 172.21.15.7:0/3366502505 shutdown_connections 2023-07-20T02:51:33.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.198+0000 7f78203ff700 1 -- 172.21.15.7:0/3366502505 wait complete. 2023-07-20T02:51:33.214 INFO:teuthology.orchestra.run.smithi007.stderr:set pool 4 allow_ec_overwrites to true 2023-07-20T02:51:33.234 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2023-07-20T02:51:33.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.427+0000 7fad2bbff700 1 -- 172.21.15.7:0/88864358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad24100760 msgr2=0x7fad24100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:33.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.427+0000 7fad2bbff700 1 --2- 172.21.15.7:0/88864358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad24100760 0x7fad24100b80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fad200099c0 tx=0x7fad20009cd0 comp rx=0 tx=0).stop 2023-07-20T02:51:33.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.427+0000 7fad2bbff700 1 -- 172.21.15.7:0/88864358 shutdown_connections 2023-07-20T02:51:33.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.427+0000 7fad2bbff700 1 --2- 172.21.15.7:0/88864358 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad241022f0 0x7fad24108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:33.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.427+0000 7fad2bbff700 1 --2- 172.21.15.7:0/88864358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad24100760 0x7fad24100b80 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:33.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.427+0000 7fad2bbff700 1 --2- 172.21.15.7:0/88864358 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad24101950 0x7fad24101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:33.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.427+0000 7fad2bbff700 1 -- 172.21.15.7:0/88864358 >> 172.21.15.7:0/88864358 conn(0x7fad240fbcb0 msgr2=0x7fad240fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:33.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.428+0000 7fad2bbff700 1 -- 172.21.15.7:0/88864358 shutdown_connections 2023-07-20T02:51:33.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.428+0000 7fad2bbff700 1 -- 172.21.15.7:0/88864358 wait complete. 2023-07-20T02:51:33.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.429+0000 7fad2bbff700 1 Processor -- start 2023-07-20T02:51:33.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.429+0000 7fad2bbff700 1 -- start start 2023-07-20T02:51:33.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.429+0000 7fad2bbff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad24100760 0x7fad24074520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:33.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.429+0000 7fad2bbff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad24101950 0x7fad24072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:33.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.429+0000 7fad2bbff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad241022f0 0x7fad240730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:33.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.429+0000 7fad2bbff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2419a380 con 0x7fad24101950 2023-07-20T02:51:33.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.429+0000 7fad2bbff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad240735e0 con 0x7fad241022f0 2023-07-20T02:51:33.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.429+0000 7fad2bbff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad240737a0 con 0x7fad24100760 2023-07-20T02:51:33.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.429+0000 7fad2919a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad24101950 0x7fad24072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:33.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.429+0000 7fad2919a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad24101950 0x7fad24072b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51854/0 (socket says 172.21.15.7:51854) 2023-07-20T02:51:33.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.429+0000 7fad2919a700 1 -- 172.21.15.7:0/2367900216 learned_addr learned my addr 172.21.15.7:0/2367900216 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:33.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.429+0000 7fad2999b700 1 --2- 172.21.15.7:0/2367900216 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad24100760 0x7fad24074520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:33.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.429+0000 7fad2a19c700 1 --2- 172.21.15.7:0/2367900216 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad241022f0 0x7fad240730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:33.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.430+0000 7fad2919a700 1 -- 172.21.15.7:0/2367900216 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad24100760 msgr2=0x7fad24074520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:33.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.430+0000 7fad2919a700 1 --2- 172.21.15.7:0/2367900216 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad24100760 0x7fad24074520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:33.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.430+0000 7fad2919a700 1 -- 172.21.15.7:0/2367900216 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad241022f0 msgr2=0x7fad240730a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:33.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.430+0000 7fad2919a700 1 --2- 172.21.15.7:0/2367900216 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad241022f0 0x7fad240730a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:33.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.430+0000 7fad2919a700 1 -- 172.21.15.7:0/2367900216 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad200095f0 con 0x7fad24101950 2023-07-20T02:51:33.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.430+0000 7fad2919a700 1 --2- 172.21.15.7:0/2367900216 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad24101950 0x7fad24072b60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fad1400e9a0 tx=0x7fad1400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:33.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.430+0000 7fad1affd700 1 -- 172.21.15.7:0/2367900216 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad14009790 con 0x7fad24101950 2023-07-20T02:51:33.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.430+0000 7fad2bbff700 1 -- 172.21.15.7:0/2367900216 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad241a4d30 con 0x7fad24101950 2023-07-20T02:51:33.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.430+0000 7fad1affd700 1 -- 172.21.15.7:0/2367900216 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fad14010430 con 0x7fad24101950 2023-07-20T02:51:33.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.430+0000 7fad1affd700 1 -- 172.21.15.7:0/2367900216 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad14018680 con 0x7fad24101950 2023-07-20T02:51:33.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.430+0000 7fad2bbff700 1 -- 172.21.15.7:0/2367900216 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad241a51b0 con 0x7fad24101950 2023-07-20T02:51:33.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.432+0000 7fad2bbff700 1 -- 172.21.15.7:0/2367900216 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad241076e0 con 0x7fad24101950 2023-07-20T02:51:33.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.433+0000 7fad1affd700 1 -- 172.21.15.7:0/2367900216 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fad140105a0 con 0x7fad24101950 2023-07-20T02:51:33.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:33 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/4260662413' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-20T02:51:33.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:33 smithi007 ceph-mon[126353]: osdmap e85: 12 total, 12 up, 12 in 2023-07-20T02:51:33.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:33 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/3366502505' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-07-20T02:51:33.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:33 smithi007 ceph-mon[126353]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-07-20T02:51:33.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.435+0000 7fad1affd700 1 --2- 172.21.15.7:0/2367900216 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fad100a11d0 0x7fad100a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:33.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.435+0000 7fad1affd700 1 -- 172.21.15.7:0/2367900216 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8698+0+0 (secure 0 0 0) 0x7fad14014070 con 0x7fad24101950 2023-07-20T02:51:33.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.435+0000 7fad2999b700 1 --2- 172.21.15.7:0/2367900216 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fad100a11d0 0x7fad100a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:33.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.436+0000 7fad2999b700 1 --2- 172.21.15.7:0/2367900216 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fad100a11d0 0x7fad100a3690 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fad20009990 tx=0x7fad20005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:33.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.439+0000 7fad1affd700 1 -- 172.21.15.7:0/2367900216 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fad14051780 con 0x7fad24101950 2023-07-20T02:51:33.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:33 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/4260662413' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-20T02:51:33.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:33 smithi188 ceph-mon[129553]: osdmap e85: 12 total, 12 up, 12 in 2023-07-20T02:51:33.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:33 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/3366502505' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-07-20T02:51:33.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:33 smithi188 ceph-mon[129553]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-07-20T02:51:33.641 INFO:journalctl@ceph.mds.b.smithi158.stdout:Jul 20 02:51:33 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-b[154225]: 2023-07-20T02:51:33.134+0000 7f091b11f700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-07-20T02:51:33.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:33 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/4260662413' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-20T02:51:33.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:33 smithi158 ceph-mon[128925]: osdmap e85: 12 total, 12 up, 12 in 2023-07-20T02:51:33.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:33 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/3366502505' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-07-20T02:51:33.642 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:33 smithi158 ceph-mon[128925]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-07-20T02:51:33.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:33.696+0000 7fad2bbff700 1 -- 172.21.15.7:0/2367900216 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7fad2402c970 con 0x7fad24101950 2023-07-20T02:51:34.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.176+0000 7fad1affd700 1 -- 172.21.15.7:0/2367900216 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) v1 ==== 137+0+0 (secure 0 0 0) 0x7fad140cee60 con 0x7fad24101950 2023-07-20T02:51:34.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.179+0000 7fad2bbff700 1 -- 172.21.15.7:0/2367900216 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fad100a11d0 msgr2=0x7fad100a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:34.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.179+0000 7fad2bbff700 1 --2- 172.21.15.7:0/2367900216 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fad100a11d0 0x7fad100a3690 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fad20009990 tx=0x7fad20005fb0 comp rx=0 tx=0).stop 2023-07-20T02:51:34.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.179+0000 7fad2bbff700 1 -- 172.21.15.7:0/2367900216 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad24101950 msgr2=0x7fad24072b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:34.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.179+0000 7fad2bbff700 1 --2- 172.21.15.7:0/2367900216 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad24101950 0x7fad24072b60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fad1400e9a0 tx=0x7fad1400ecb0 comp rx=0 tx=0).stop 2023-07-20T02:51:34.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.180+0000 7fad2bbff700 1 -- 172.21.15.7:0/2367900216 shutdown_connections 2023-07-20T02:51:34.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.180+0000 7fad2bbff700 1 --2- 172.21.15.7:0/2367900216 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fad241022f0 0x7fad240730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:34.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.180+0000 7fad2bbff700 1 --2- 172.21.15.7:0/2367900216 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad24100760 0x7fad24074520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:34.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.180+0000 7fad2bbff700 1 --2- 172.21.15.7:0/2367900216 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fad100a11d0 0x7fad100a3690 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:34.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.180+0000 7fad2bbff700 1 --2- 172.21.15.7:0/2367900216 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad24101950 0x7fad24072b60 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:34.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.180+0000 7fad2bbff700 1 -- 172.21.15.7:0/2367900216 >> 172.21.15.7:0/2367900216 conn(0x7fad240fbcb0 msgr2=0x7fad240fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:34.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.180+0000 7fad2bbff700 1 -- 172.21.15.7:0/2367900216 shutdown_connections 2023-07-20T02:51:34.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.180+0000 7fad2bbff700 1 -- 172.21.15.7:0/2367900216 wait complete. 2023-07-20T02:51:34.196 INFO:teuthology.orchestra.run.smithi007.stderr:added data pool 4 to fsmap 2023-07-20T02:51:34.215 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T02:51:34.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.409+0000 7ff57fdff700 1 -- 172.21.15.7:0/905015311 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff578101960 msgr2=0x7ff578101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.409+0000 7ff57fdff700 1 --2- 172.21.15.7:0/905015311 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff578101960 0x7ff578101dc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff5680099c0 tx=0x7ff568009cd0 comp rx=0 tx=0).stop 2023-07-20T02:51:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.409+0000 7ff57fdff700 1 -- 172.21.15.7:0/905015311 shutdown_connections 2023-07-20T02:51:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.409+0000 7ff57fdff700 1 --2- 172.21.15.7:0/905015311 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff578102300 0x7ff578108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.409+0000 7ff57fdff700 1 --2- 172.21.15.7:0/905015311 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff578100770 0x7ff578100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.409+0000 7ff57fdff700 1 --2- 172.21.15.7:0/905015311 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff578101960 0x7ff578101dc0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.409+0000 7ff57fdff700 1 -- 172.21.15.7:0/905015311 >> 172.21.15.7:0/905015311 conn(0x7ff5780fbca0 msgr2=0x7ff5780fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:34.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.409+0000 7ff57fdff700 1 -- 172.21.15.7:0/905015311 shutdown_connections 2023-07-20T02:51:34.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.409+0000 7ff57fdff700 1 -- 172.21.15.7:0/905015311 wait complete. 2023-07-20T02:51:34.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.410+0000 7ff57fdff700 1 Processor -- start 2023-07-20T02:51:34.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.410+0000 7ff57fdff700 1 -- start start 2023-07-20T02:51:34.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.410+0000 7ff57fdff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff578100770 0x7ff578196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:34.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.410+0000 7ff57fdff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff578101960 0x7ff578196970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:34.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.410+0000 7ff57fdff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff578102300 0x7ff57819d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:34.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.410+0000 7ff57fdff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57819dfb0 con 0x7ff578101960 2023-07-20T02:51:34.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.410+0000 7ff57fdff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57819e0f0 con 0x7ff578100770 2023-07-20T02:51:34.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.410+0000 7ff57fdff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57819e280 con 0x7ff578102300 2023-07-20T02:51:34.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.411+0000 7ff57d39a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff578101960 0x7ff578196970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:34.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.411+0000 7ff57d39a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff578101960 0x7ff578196970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51860/0 (socket says 172.21.15.7:51860) 2023-07-20T02:51:34.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.411+0000 7ff57d39a700 1 -- 172.21.15.7:0/665950530 learned_addr learned my addr 172.21.15.7:0/665950530 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:34.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.411+0000 7ff57db9b700 1 --2- 172.21.15.7:0/665950530 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff578100770 0x7ff578196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:34.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.411+0000 7ff57e39c700 1 --2- 172.21.15.7:0/665950530 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff578102300 0x7ff57819d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:34.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.411+0000 7ff57d39a700 1 -- 172.21.15.7:0/665950530 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff578102300 msgr2=0x7ff57819d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:34.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.411+0000 7ff57d39a700 1 --2- 172.21.15.7:0/665950530 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff578102300 0x7ff57819d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:34.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.411+0000 7ff57d39a700 1 -- 172.21.15.7:0/665950530 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff578100770 msgr2=0x7ff578196430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:34.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.411+0000 7ff57d39a700 1 --2- 172.21.15.7:0/665950530 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff578100770 0x7ff578196430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:34.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.411+0000 7ff57d39a700 1 -- 172.21.15.7:0/665950530 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5680095f0 con 0x7ff578101960 2023-07-20T02:51:34.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.411+0000 7ff57d39a700 1 --2- 172.21.15.7:0/665950530 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff578101960 0x7ff578196970 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff568009990 tx=0x7ff56800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:34.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.411+0000 7ff56effd700 1 -- 172.21.15.7:0/665950530 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56801d070 con 0x7ff578101960 2023-07-20T02:51:34.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.411+0000 7ff56effd700 1 -- 172.21.15.7:0/665950530 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff5680054f0 con 0x7ff578101960 2023-07-20T02:51:34.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.412+0000 7ff57fdff700 1 -- 172.21.15.7:0/665950530 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff57819e500 con 0x7ff578101960 2023-07-20T02:51:34.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.412+0000 7ff56effd700 1 -- 172.21.15.7:0/665950530 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff568004b00 con 0x7ff578101960 2023-07-20T02:51:34.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.412+0000 7ff57fdff700 1 -- 172.21.15.7:0/665950530 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff57819e980 con 0x7ff578101960 2023-07-20T02:51:34.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.413+0000 7ff57fdff700 1 -- 172.21.15.7:0/665950530 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff57810bfe0 con 0x7ff578101960 2023-07-20T02:51:34.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.414+0000 7ff56effd700 1 -- 172.21.15.7:0/665950530 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7ff56800fb80 con 0x7ff578101960 2023-07-20T02:51:34.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.416+0000 7ff56effd700 1 --2- 172.21.15.7:0/665950530 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7ff5640a1220 0x7ff5640a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:34.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.416+0000 7ff56effd700 1 -- 172.21.15.7:0/665950530 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff5680cd890 con 0x7ff578101960 2023-07-20T02:51:34.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.416+0000 7ff57db9b700 1 --2- 172.21.15.7:0/665950530 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7ff5640a1220 0x7ff5640a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:34.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.417+0000 7ff57db9b700 1 --2- 172.21.15.7:0/665950530 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7ff5640a1220 0x7ff5640a36e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff5781017c0 tx=0x7ff574009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:34.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.419+0000 7ff56effd700 1 -- 172.21.15.7:0/665950530 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff568096e20 con 0x7ff578101960 2023-07-20T02:51:34.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:34 smithi007 ceph-mon[126353]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-07-20T02:51:34.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:34 smithi007 ceph-mon[126353]: osdmap e86: 12 total, 12 up, 12 in 2023-07-20T02:51:34.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:34 smithi007 ceph-mon[126353]: pgmap v276: 129 pgs: 97 active+clean, 32 unknown; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2023-07-20T02:51:34.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:34 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/2367900216' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-07-20T02:51:34.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:34 smithi188 ceph-mon[129553]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-07-20T02:51:34.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:34 smithi188 ceph-mon[129553]: osdmap e86: 12 total, 12 up, 12 in 2023-07-20T02:51:34.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:34 smithi188 ceph-mon[129553]: pgmap v276: 129 pgs: 97 active+clean, 32 unknown; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2023-07-20T02:51:34.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:34 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/2367900216' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-07-20T02:51:34.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:34 smithi158 ceph-mon[128925]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-07-20T02:51:34.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:34 smithi158 ceph-mon[128925]: osdmap e86: 12 total, 12 up, 12 in 2023-07-20T02:51:34.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:34 smithi158 ceph-mon[128925]: pgmap v276: 129 pgs: 97 active+clean, 32 unknown; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2023-07-20T02:51:34.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:34 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/2367900216' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-07-20T02:51:34.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.674+0000 7ff57fdff700 1 -- 172.21.15.7:0/665950530 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff57810c200 con 0x7ff578101960 2023-07-20T02:51:34.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.675+0000 7ff56effd700 1 -- 172.21.15.7:0/665950530 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+10757 (secure 0 0 0) 0x7ff56809a440 con 0x7ff578101960 2023-07-20T02:51:34.689 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:51:34.689 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/190105946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":190105946},{"type":"v1","addr":"172.21.15.7:6837","nonce":190105946}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/3310846795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":3310846795},{"type":"v1","addr":"172.21.15.7:6839","nonce":3310846795}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14568,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3189534126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3189534126},{"type":"v1","addr":"172.21.15.7:6841","nonce":3189534126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/1435841351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":1435841351},{"type":"v1","addr":"172.21.15.188:6833","nonce":1435841351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/401861265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":401861265},{"type":"v1","addr":"172.21.15.158:6833","nonce":401861265}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/4064058311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":4064058311},{"type":"v1","addr":"172.21.15.7:6835","nonce":4064058311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3931487788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3931487788},{"type":"v1","addr":"172.21.15.158:6835","nonce":3931487788}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/857155289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":857155289},{"type":"v1","addr":"172.21.15.188:6835","nonce":857155289}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1032497949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6836","nonce":1032497949},{"type":"v1","addr":"172.21.15.158:6837","nonce":1032497949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/775436728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":775436728},{"type":"v1","addr":"172.21.15.188:6837","nonce":775436728}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24562,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6839/450074084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":450074084},{"type":"v1","addr":"172.21.15.188:6839","nonce":450074084}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-07-20T02:51:28.640448+0000","modified":"2023-07-20T02:51:34.171163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6839/2552719389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6838","nonce":2552719389},{"type":"v1","addr":"172.21.15.158:6839","nonce":2552719389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-07-20T02:51:34.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.678+0000 7ff57fdff700 1 -- 172.21.15.7:0/665950530 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7ff5640a1220 msgr2=0x7ff5640a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:34.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.678+0000 7ff57fdff700 1 --2- 172.21.15.7:0/665950530 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7ff5640a1220 0x7ff5640a36e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff5781017c0 tx=0x7ff574009260 comp rx=0 tx=0).stop 2023-07-20T02:51:34.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.678+0000 7ff57fdff700 1 -- 172.21.15.7:0/665950530 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff578101960 msgr2=0x7ff578196970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:34.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.678+0000 7ff57fdff700 1 --2- 172.21.15.7:0/665950530 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff578101960 0x7ff578196970 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff568009990 tx=0x7ff56800f690 comp rx=0 tx=0).stop 2023-07-20T02:51:34.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.678+0000 7ff57fdff700 1 -- 172.21.15.7:0/665950530 shutdown_connections 2023-07-20T02:51:34.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.678+0000 7ff57fdff700 1 --2- 172.21.15.7:0/665950530 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff578102300 0x7ff57819d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:34.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.678+0000 7ff57fdff700 1 --2- 172.21.15.7:0/665950530 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7ff578100770 0x7ff578196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:34.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.678+0000 7ff57fdff700 1 --2- 172.21.15.7:0/665950530 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7ff5640a1220 0x7ff5640a36e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:34.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.678+0000 7ff57fdff700 1 --2- 172.21.15.7:0/665950530 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff578101960 0x7ff578196970 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:34.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.678+0000 7ff57fdff700 1 -- 172.21.15.7:0/665950530 >> 172.21.15.7:0/665950530 conn(0x7ff5780fbca0 msgr2=0x7ff5780fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:34.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.678+0000 7ff57fdff700 1 -- 172.21.15.7:0/665950530 shutdown_connections 2023-07-20T02:51:34.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.678+0000 7ff57fdff700 1 -- 172.21.15.7:0/665950530 wait complete. 2023-07-20T02:51:34.694 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 17 2023-07-20T02:51:34.710 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T02:51:34.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.900+0000 7f56019ff700 1 -- 172.21.15.7:0/781409225 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55fc073170 msgr2=0x7f55fc0735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:34.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.900+0000 7f56019ff700 1 --2- 172.21.15.7:0/781409225 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55fc073170 0x7f55fc0735f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f55f0009990 tx=0x7f55f0009ca0 comp rx=0 tx=0).stop 2023-07-20T02:51:34.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.900+0000 7f56019ff700 1 -- 172.21.15.7:0/781409225 shutdown_connections 2023-07-20T02:51:34.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.900+0000 7f56019ff700 1 --2- 172.21.15.7:0/781409225 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55fc108680 0x7f55fc10aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:34.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.900+0000 7f56019ff700 1 --2- 172.21.15.7:0/781409225 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55fc074750 0x7f55fc072ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:34.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.901+0000 7f56019ff700 1 --2- 172.21.15.7:0/781409225 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55fc073170 0x7f55fc0735f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:34.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.901+0000 7f56019ff700 1 -- 172.21.15.7:0/781409225 >> 172.21.15.7:0/781409225 conn(0x7f55fc0fbd20 msgr2=0x7f55fc0fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:34.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.901+0000 7f56019ff700 1 -- 172.21.15.7:0/781409225 shutdown_connections 2023-07-20T02:51:34.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.901+0000 7f56019ff700 1 -- 172.21.15.7:0/781409225 wait complete. 2023-07-20T02:51:34.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.902+0000 7f56019ff700 1 Processor -- start 2023-07-20T02:51:34.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.902+0000 7f56019ff700 1 -- start start 2023-07-20T02:51:34.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.902+0000 7f56019ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55fc073170 0x7f55fc1985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:34.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.902+0000 7f56019ff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55fc074750 0x7f55fc198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:34.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.902+0000 7f56019ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55fc108680 0x7f55fc19fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:34.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.902+0000 7f56019ff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55fc1a0160 con 0x7f55fc108680 2023-07-20T02:51:34.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.902+0000 7f56019ff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55fc1a02a0 con 0x7f55fc074750 2023-07-20T02:51:34.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.902+0000 7f56019ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55fc1a0430 con 0x7f55fc073170 2023-07-20T02:51:34.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.902+0000 7f55fb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55fc108680 0x7f55fc19fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:34.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.902+0000 7f55faffd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55fc073170 0x7f55fc1985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:34.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.902+0000 7f55fa7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55fc074750 0x7f55fc198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:34.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.902+0000 7f55fa7fc700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55fc074750 0x7f55fc198b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.7:60578/0 (socket says 172.21.15.7:60578) 2023-07-20T02:51:34.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.902+0000 7f55fb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55fc108680 0x7f55fc19fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51884/0 (socket says 172.21.15.7:51884) 2023-07-20T02:51:34.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.902+0000 7f55faffd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55fc073170 0x7f55fc1985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.7:52674/0 (socket says 172.21.15.7:52674) 2023-07-20T02:51:34.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.902+0000 7f55fa7fc700 1 -- 172.21.15.7:0/3674528279 learned_addr learned my addr 172.21.15.7:0/3674528279 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:34.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.903+0000 7f55faffd700 1 -- 172.21.15.7:0/3674528279 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55fc074750 msgr2=0x7f55fc198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:34.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.903+0000 7f55faffd700 1 --2- 172.21.15.7:0/3674528279 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55fc074750 0x7f55fc198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:34.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.903+0000 7f55faffd700 1 -- 172.21.15.7:0/3674528279 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55fc108680 msgr2=0x7f55fc19fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:34.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.903+0000 7f55faffd700 1 --2- 172.21.15.7:0/3674528279 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55fc108680 0x7f55fc19fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:34.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.903+0000 7f55faffd700 1 -- 172.21.15.7:0/3674528279 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f00095f0 con 0x7f55fc073170 2023-07-20T02:51:34.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.903+0000 7f55fa7fc700 1 --2- 172.21.15.7:0/3674528279 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55fc074750 0x7f55fc198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T02:51:34.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.903+0000 7f55fb7fe700 1 --2- 172.21.15.7:0/3674528279 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55fc108680 0x7f55fc19fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T02:51:34.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.903+0000 7f55faffd700 1 --2- 172.21.15.7:0/3674528279 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55fc073170 0x7f55fc1985e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f55ec00e9f0 tx=0x7f55ec00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:34.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.903+0000 7f56009fd700 1 -- 172.21.15.7:0/3674528279 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ec00c9c0 con 0x7f55fc073170 2023-07-20T02:51:34.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.903+0000 7f56019ff700 1 -- 172.21.15.7:0/3674528279 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55fc1a0710 con 0x7f55fc073170 2023-07-20T02:51:34.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.904+0000 7f56009fd700 1 -- 172.21.15.7:0/3674528279 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f55ec010430 con 0x7f55fc073170 2023-07-20T02:51:34.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.904+0000 7f56009fd700 1 -- 172.21.15.7:0/3674528279 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ec018680 con 0x7f55fc073170 2023-07-20T02:51:34.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.904+0000 7f56019ff700 1 -- 172.21.15.7:0/3674528279 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55fc1a0c70 con 0x7f55fc073170 2023-07-20T02:51:34.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.905+0000 7f56019ff700 1 -- 172.21.15.7:0/3674528279 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55fc107490 con 0x7f55fc073170 2023-07-20T02:51:34.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.907+0000 7f56009fd700 1 -- 172.21.15.7:0/3674528279 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f55ec010860 con 0x7f55fc073170 2023-07-20T02:51:34.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.908+0000 7f56009fd700 1 --2- 172.21.15.7:0/3674528279 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f55e40a11a0 0x7f55e40a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:34.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.908+0000 7f56009fd700 1 -- 172.21.15.7:0/3674528279 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f55ec014070 con 0x7f55fc073170 2023-07-20T02:51:34.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.908+0000 7f55fa7fc700 1 --2- 172.21.15.7:0/3674528279 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f55e40a11a0 0x7f55e40a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:34.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.909+0000 7f55fa7fc700 1 --2- 172.21.15.7:0/3674528279 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f55e40a11a0 0x7f55e40a3660 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f55f000b2e0 tx=0x7f55f0005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:34.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:34.910+0000 7f56009fd700 1 -- 172.21.15.7:0/3674528279 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f55ec096340 con 0x7f55fc073170 2023-07-20T02:51:35.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.120+0000 7f56019ff700 1 -- 172.21.15.7:0/3674528279 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f55fc02cc20 con 0x7f55fc073170 2023-07-20T02:51:35.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.122+0000 7f56009fd700 1 -- 172.21.15.7:0/3674528279 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+23927 (secure 0 0 0) 0x7f55ec099960 con 0x7f55fc073170 2023-07-20T02:51:35.136 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:51:35.136 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":87,"fsid":"1b04643c-26a7-11ee-9b34-001a4aab830c","created":"2023-07-20T02:43:56.544719+0000","modified":"2023-07-20T02:51:33.697865+0000","last_up_change":"2023-07-20T02:50:40.126926+0000","last_in_change":"2023-07-20T02:50:21.875020+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T02:47:12.055539+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-20T02:51:26.617961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-20T02:51:28.081016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-20T02:51:31.658088+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ad6c6339-6e23-4c21-9fde-2dbecdafdb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6803","nonce":3322992091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6805","nonce":3322992091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6809","nonce":3322992091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6807","nonce":3322992091}]},"public_addr":"172.21.15.7:6803/3322992091","cluster_addr":"172.21.15.7:6805/3322992091","heartbeat_back_addr":"172.21.15.7:6809/3322992091","heartbeat_front_addr":"172.21.15.7:6807/3322992091","state":["exists","up"]},{"osd":1,"uuid":"cad207b3-a1ea-4f07-bd94-9b11fd573a69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6801","nonce":2796771612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6803","nonce":2796771612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6807","nonce":2796771612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6805","nonce":2796771612}]},"public_addr":"172.21.15.158:6801/2796771612","cluster_addr":"172.21.15.158:6803/2796771612","heartbeat_back_addr":"172.21.15.158:6807/2796771612","heartbeat_front_addr":"172.21.15.158:6805/2796771612","state":["exists","up"]},{"osd":2,"uuid":"2c777fe6-d003-49d0-af37-f63fca37ff06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6801","nonce":2928307258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6803","nonce":2928307258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6807","nonce":2928307258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6805","nonce":2928307258}]},"public_addr":"172.21.15.188:6801/2928307258","cluster_addr":"172.21.15.188:6803/2928307258","heartbeat_back_addr":"172.21.15.188:6807/2928307258","heartbeat_front_addr":"172.21.15.188:6805/2928307258","state":["exists","up"]},{"osd":3,"uuid":"59e29965-018f-4eb6-adbd-1a9e1df84245","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6811","nonce":2192953827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6813","nonce":2192953827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6817","nonce":2192953827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6815","nonce":2192953827}]},"public_addr":"172.21.15.7:6811/2192953827","cluster_addr":"172.21.15.7:6813/2192953827","heartbeat_back_addr":"172.21.15.7:6817/2192953827","heartbeat_front_addr":"172.21.15.7:6815/2192953827","state":["exists","up"]},{"osd":4,"uuid":"e2453384-3b09-4f2a-bea7-172d8fc466d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6809","nonce":3330591132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6810","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6811","nonce":3330591132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6814","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6815","nonce":3330591132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6813","nonce":3330591132}]},"public_addr":"172.21.15.158:6809/3330591132","cluster_addr":"172.21.15.158:6811/3330591132","heartbeat_back_addr":"172.21.15.158:6815/3330591132","heartbeat_front_addr":"172.21.15.158:6813/3330591132","state":["exists","up"]},{"osd":5,"uuid":"c3c04c13-0ce5-440d-9e62-0e301ad9f771","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6809","nonce":3142012805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6811","nonce":3142012805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6815","nonce":3142012805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6813","nonce":3142012805}]},"public_addr":"172.21.15.188:6809/3142012805","cluster_addr":"172.21.15.188:6811/3142012805","heartbeat_back_addr":"172.21.15.188:6815/3142012805","heartbeat_front_addr":"172.21.15.188:6813/3142012805","state":["exists","up"]},{"osd":6,"uuid":"82ef2d33-1045-41f1-bf69-b03404bcace8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6819","nonce":859560576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6821","nonce":859560576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6825","nonce":859560576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6823","nonce":859560576}]},"public_addr":"172.21.15.7:6819/859560576","cluster_addr":"172.21.15.7:6821/859560576","heartbeat_back_addr":"172.21.15.7:6825/859560576","heartbeat_front_addr":"172.21.15.7:6823/859560576","state":["exists","up"]},{"osd":7,"uuid":"020ec859-3fc2-4da2-a0dc-10530a4a366c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6817","nonce":2578758635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6819","nonce":2578758635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6822","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6823","nonce":2578758635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6821","nonce":2578758635}]},"public_addr":"172.21.15.158:6817/2578758635","cluster_addr":"172.21.15.158:6819/2578758635","heartbeat_back_addr":"172.21.15.158:6823/2578758635","heartbeat_front_addr":"172.21.15.158:6821/2578758635","state":["exists","up"]},{"osd":8,"uuid":"4081496a-45ee-4b25-8f17-a23665521187","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6817","nonce":3020824843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6819","nonce":3020824843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6823","nonce":3020824843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6821","nonce":3020824843}]},"public_addr":"172.21.15.188:6817/3020824843","cluster_addr":"172.21.15.188:6819/3020824843","heartbeat_back_addr":"172.21.15.188:6823/3020824843","heartbeat_front_addr":"172.21.15.188:6821/3020824843","state":["exists","up"]},{"osd":9,"uuid":"723580ae-efe3-4d86-95fd-d7a4c0631425","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6827","nonce":1612464825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6829","nonce":1612464825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6833","nonce":1612464825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6831","nonce":1612464825}]},"public_addr":"172.21.15.7:6827/1612464825","cluster_addr":"172.21.15.7:6829/1612464825","heartbeat_back_addr":"172.21.15.7:6833/1612464825","heartbeat_front_addr":"172.21.15.7:6831/1612464825","state":["exists","up"]},{"osd":10,"uuid":"c2651548-fe46-4d2d-b911-278036412ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6825","nonce":1653166210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6827","nonce":1653166210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6831","nonce":1653166210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6829","nonce":1653166210}]},"public_addr":"172.21.15.158:6825/1653166210","cluster_addr":"172.21.15.158:6827/1653166210","heartbeat_back_addr":"172.21.15.158:6831/1653166210","heartbeat_front_addr":"172.21.15.158:6829/1653166210","state":["exists","up"]},{"osd":11,"uuid":"f7262bcb-4243-403a-a321-fc2bdf1d4d25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6825","nonce":1230196525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6827","nonce":1230196525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6831","nonce":1230196525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6829","nonce":1230196525}]},"public_addr":"172.21.15.188:6825/1230196525","cluster_addr":"172.21.15.188:6827/1230196525","heartbeat_back_addr":"172.21.15.188:6831/1230196525","heartbeat_front_addr":"172.21.15.188:6829/1230196525","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:46:26.803832+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:46:48.726253+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:47:09.744123+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:47:36.907588+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:47:57.696685+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:48:19.120806+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:48:45.313043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:49:06.686384+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:49:28.475078+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:49:54.984033+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:50:16.541259+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:50:38.221746+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/3892248290":"2023-07-21T02:44:53.458649+0000","172.21.15.7:0/2373058722":"2023-07-21T02:44:53.458649+0000","172.21.15.7:6801/2926310425":"2023-07-21T02:44:53.458649+0000","172.21.15.7:0/4222569264":"2023-07-21T02:44:25.804258+0000","172.21.15.7:0/1357065986":"2023-07-21T02:44:25.804258+0000","172.21.15.7:0/884368977":"2023-07-21T02:44:53.458649+0000","172.21.15.7:6801/1224472946":"2023-07-21T02:44:25.804258+0000","172.21.15.7:0/1459293748":"2023-07-21T02:44:25.804258+0000","172.21.15.7:6800/1224472946":"2023-07-21T02:44:25.804258+0000","172.21.15.7:6800/2926310425":"2023-07-21T02:44:53.458649+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T02:51:35.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.124+0000 7f56019ff700 1 -- 172.21.15.7:0/3674528279 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f55e40a11a0 msgr2=0x7f55e40a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:35.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.124+0000 7f56019ff700 1 --2- 172.21.15.7:0/3674528279 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f55e40a11a0 0x7f55e40a3660 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f55f000b2e0 tx=0x7f55f0005960 comp rx=0 tx=0).stop 2023-07-20T02:51:35.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.124+0000 7f56019ff700 1 -- 172.21.15.7:0/3674528279 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55fc073170 msgr2=0x7f55fc1985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:35.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.125+0000 7f56019ff700 1 --2- 172.21.15.7:0/3674528279 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55fc073170 0x7f55fc1985e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f55ec00e9f0 tx=0x7f55ec00ed00 comp rx=0 tx=0).stop 2023-07-20T02:51:35.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.125+0000 7f56019ff700 1 -- 172.21.15.7:0/3674528279 shutdown_connections 2023-07-20T02:51:35.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.125+0000 7f56019ff700 1 --2- 172.21.15.7:0/3674528279 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f55e40a11a0 0x7f55e40a3660 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:35.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.125+0000 7f56019ff700 1 --2- 172.21.15.7:0/3674528279 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55fc108680 0x7f55fc19fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:35.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.125+0000 7f56019ff700 1 --2- 172.21.15.7:0/3674528279 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55fc073170 0x7f55fc1985e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:35.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.125+0000 7f56019ff700 1 --2- 172.21.15.7:0/3674528279 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f55fc074750 0x7f55fc198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:35.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.125+0000 7f56019ff700 1 -- 172.21.15.7:0/3674528279 >> 172.21.15.7:0/3674528279 conn(0x7f55fc0fbd20 msgr2=0x7f55fc1045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:35.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.125+0000 7f56019ff700 1 -- 172.21.15.7:0/3674528279 shutdown_connections 2023-07-20T02:51:35.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.125+0000 7f56019ff700 1 -- 172.21.15.7:0/3674528279 wait complete. 2023-07-20T02:51:35.158 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T02:51:35.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.350+0000 7fabe29ff700 1 -- 172.21.15.7:0/1872478766 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fabdc100770 msgr2=0x7fabdc100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:35.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.350+0000 7fabe29ff700 1 --2- 172.21.15.7:0/1872478766 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fabdc100770 0x7fabdc100b90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fabcc0098f0 tx=0x7fabcc009c00 comp rx=0 tx=0).stop 2023-07-20T02:51:35.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.351+0000 7fabe29ff700 1 -- 172.21.15.7:0/1872478766 shutdown_connections 2023-07-20T02:51:35.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.351+0000 7fabe29ff700 1 --2- 172.21.15.7:0/1872478766 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabdc102300 0x7fabdc108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:35.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.351+0000 7fabe29ff700 1 --2- 172.21.15.7:0/1872478766 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fabdc100770 0x7fabdc100b90 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:35.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.351+0000 7fabe29ff700 1 --2- 172.21.15.7:0/1872478766 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabdc101960 0x7fabdc101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:35.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.351+0000 7fabe29ff700 1 -- 172.21.15.7:0/1872478766 >> 172.21.15.7:0/1872478766 conn(0x7fabdc0fbca0 msgr2=0x7fabdc0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:35.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.351+0000 7fabe29ff700 1 -- 172.21.15.7:0/1872478766 shutdown_connections 2023-07-20T02:51:35.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.351+0000 7fabe29ff700 1 -- 172.21.15.7:0/1872478766 wait complete. 2023-07-20T02:51:35.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.352+0000 7fabe29ff700 1 Processor -- start 2023-07-20T02:51:35.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.352+0000 7fabe29ff700 1 -- start start 2023-07-20T02:51:35.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.352+0000 7fabe29ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabdc100770 0x7fabdc1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:35.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.352+0000 7fabe29ff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabdc101960 0x7fabdc198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:35.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.352+0000 7fabe29ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fabdc102300 0x7fabdc19fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:35.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.352+0000 7fabe29ff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabdc1a0240 con 0x7fabdc100770 2023-07-20T02:51:35.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.352+0000 7fabe29ff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabdc1a0380 con 0x7fabdc101960 2023-07-20T02:51:35.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.352+0000 7fabe29ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabdc1a0510 con 0x7fabdc102300 2023-07-20T02:51:35.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.352+0000 7fabdbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabdc100770 0x7fabdc1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:35.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.352+0000 7fabdbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabdc100770 0x7fabdc1986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51902/0 (socket says 172.21.15.7:51902) 2023-07-20T02:51:35.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.352+0000 7fabdbfff700 1 -- 172.21.15.7:0/3668575776 learned_addr learned my addr 172.21.15.7:0/3668575776 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:35.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.353+0000 7fabdb7fe700 1 --2- 172.21.15.7:0/3668575776 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabdc101960 0x7fabdc198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:35.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.353+0000 7fabe0f9c700 1 --2- 172.21.15.7:0/3668575776 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fabdc102300 0x7fabdc19fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:35.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.353+0000 7fabdbfff700 1 -- 172.21.15.7:0/3668575776 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fabdc102300 msgr2=0x7fabdc19fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:35.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.353+0000 7fabdbfff700 1 --2- 172.21.15.7:0/3668575776 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fabdc102300 0x7fabdc19fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:35.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.353+0000 7fabdbfff700 1 -- 172.21.15.7:0/3668575776 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabdc101960 msgr2=0x7fabdc198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:35.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.353+0000 7fabdbfff700 1 --2- 172.21.15.7:0/3668575776 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabdc101960 0x7fabdc198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:35.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.353+0000 7fabdbfff700 1 -- 172.21.15.7:0/3668575776 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd4009d00 con 0x7fabdc100770 2023-07-20T02:51:35.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.353+0000 7fabe0f9c700 1 --2- 172.21.15.7:0/3668575776 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fabdc102300 0x7fabdc19fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T02:51:35.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.353+0000 7fabdbfff700 1 --2- 172.21.15.7:0/3668575776 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabdc100770 0x7fabdc1986c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fabcc009fd0 tx=0x7fabcc00b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:35.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.353+0000 7fabd97fa700 1 -- 172.21.15.7:0/3668575776 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabcc01c070 con 0x7fabdc100770 2023-07-20T02:51:35.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.353+0000 7fabd97fa700 1 -- 172.21.15.7:0/3668575776 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fabcc005040 con 0x7fabdc100770 2023-07-20T02:51:35.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.353+0000 7fabd97fa700 1 -- 172.21.15.7:0/3668575776 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabcc003aa0 con 0x7fabdc100770 2023-07-20T02:51:35.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.353+0000 7fabe29ff700 1 -- 172.21.15.7:0/3668575776 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabcc009520 con 0x7fabdc100770 2023-07-20T02:51:35.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.354+0000 7fabe29ff700 1 -- 172.21.15.7:0/3668575776 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabdc1a0a20 con 0x7fabdc100770 2023-07-20T02:51:35.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.354+0000 7fabe29ff700 1 -- 172.21.15.7:0/3668575776 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabdc192dc0 con 0x7fabdc100770 2023-07-20T02:51:35.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.356+0000 7fabd97fa700 1 -- 172.21.15.7:0/3668575776 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fabcc020030 con 0x7fabdc100770 2023-07-20T02:51:35.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.358+0000 7fabd97fa700 1 --2- 172.21.15.7:0/3668575776 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fabc40a1220 0x7fabc40a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:35.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.358+0000 7fabd97fa700 1 -- 172.21.15.7:0/3668575776 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fabcc0cddc0 con 0x7fabdc100770 2023-07-20T02:51:35.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.358+0000 7fabdb7fe700 1 --2- 172.21.15.7:0/3668575776 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fabc40a1220 0x7fabc40a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:35.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.359+0000 7fabdb7fe700 1 --2- 172.21.15.7:0/3668575776 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fabc40a1220 0x7fabc40a36e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fabd0009c60 tx=0x7fabd0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:35.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.360+0000 7fabd97fa700 1 -- 172.21.15.7:0/3668575776 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fabcc0973b0 con 0x7fabdc100770 2023-07-20T02:51:35.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:35 smithi007 ceph-mon[126353]: osdmap e87: 12 total, 12 up, 12 in 2023-07-20T02:51:35.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:35 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/2367900216' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-07-20T02:51:35.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:35 smithi007 ceph-mon[126353]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T02:51:35.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:35 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/665950530' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:35.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:35 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/3674528279' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:51:35.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:35 smithi188 ceph-mon[129553]: osdmap e87: 12 total, 12 up, 12 in 2023-07-20T02:51:35.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:35 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/2367900216' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-07-20T02:51:35.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:35 smithi188 ceph-mon[129553]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T02:51:35.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:35 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/665950530' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:35.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:35 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/3674528279' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:51:35.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.571+0000 7fabe29ff700 1 -- 172.21.15.7:0/3668575776 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fabdc02c970 con 0x7fabdc100770 2023-07-20T02:51:35.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.573+0000 7fabd97fa700 1 -- 172.21.15.7:0/3668575776 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+23927 (secure 0 0 0) 0x7fabcc09a9d0 con 0x7fabdc100770 2023-07-20T02:51:35.587 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:51:35.588 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":87,"fsid":"1b04643c-26a7-11ee-9b34-001a4aab830c","created":"2023-07-20T02:43:56.544719+0000","modified":"2023-07-20T02:51:33.697865+0000","last_up_change":"2023-07-20T02:50:40.126926+0000","last_in_change":"2023-07-20T02:50:21.875020+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T02:47:12.055539+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-20T02:51:26.617961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-20T02:51:28.081016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-20T02:51:31.658088+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ad6c6339-6e23-4c21-9fde-2dbecdafdb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6803","nonce":3322992091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6805","nonce":3322992091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6809","nonce":3322992091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6807","nonce":3322992091}]},"public_addr":"172.21.15.7:6803/3322992091","cluster_addr":"172.21.15.7:6805/3322992091","heartbeat_back_addr":"172.21.15.7:6809/3322992091","heartbeat_front_addr":"172.21.15.7:6807/3322992091","state":["exists","up"]},{"osd":1,"uuid":"cad207b3-a1ea-4f07-bd94-9b11fd573a69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6801","nonce":2796771612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6803","nonce":2796771612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6807","nonce":2796771612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6805","nonce":2796771612}]},"public_addr":"172.21.15.158:6801/2796771612","cluster_addr":"172.21.15.158:6803/2796771612","heartbeat_back_addr":"172.21.15.158:6807/2796771612","heartbeat_front_addr":"172.21.15.158:6805/2796771612","state":["exists","up"]},{"osd":2,"uuid":"2c777fe6-d003-49d0-af37-f63fca37ff06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6801","nonce":2928307258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6803","nonce":2928307258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6807","nonce":2928307258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6805","nonce":2928307258}]},"public_addr":"172.21.15.188:6801/2928307258","cluster_addr":"172.21.15.188:6803/2928307258","heartbeat_back_addr":"172.21.15.188:6807/2928307258","heartbeat_front_addr":"172.21.15.188:6805/2928307258","state":["exists","up"]},{"osd":3,"uuid":"59e29965-018f-4eb6-adbd-1a9e1df84245","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6811","nonce":2192953827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6813","nonce":2192953827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6817","nonce":2192953827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6815","nonce":2192953827}]},"public_addr":"172.21.15.7:6811/2192953827","cluster_addr":"172.21.15.7:6813/2192953827","heartbeat_back_addr":"172.21.15.7:6817/2192953827","heartbeat_front_addr":"172.21.15.7:6815/2192953827","state":["exists","up"]},{"osd":4,"uuid":"e2453384-3b09-4f2a-bea7-172d8fc466d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6809","nonce":3330591132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6810","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6811","nonce":3330591132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6814","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6815","nonce":3330591132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6813","nonce":3330591132}]},"public_addr":"172.21.15.158:6809/3330591132","cluster_addr":"172.21.15.158:6811/3330591132","heartbeat_back_addr":"172.21.15.158:6815/3330591132","heartbeat_front_addr":"172.21.15.158:6813/3330591132","state":["exists","up"]},{"osd":5,"uuid":"c3c04c13-0ce5-440d-9e62-0e301ad9f771","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6809","nonce":3142012805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6811","nonce":3142012805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6815","nonce":3142012805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6813","nonce":3142012805}]},"public_addr":"172.21.15.188:6809/3142012805","cluster_addr":"172.21.15.188:6811/3142012805","heartbeat_back_addr":"172.21.15.188:6815/3142012805","heartbeat_front_addr":"172.21.15.188:6813/3142012805","state":["exists","up"]},{"osd":6,"uuid":"82ef2d33-1045-41f1-bf69-b03404bcace8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6819","nonce":859560576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6821","nonce":859560576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6825","nonce":859560576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6823","nonce":859560576}]},"public_addr":"172.21.15.7:6819/859560576","cluster_addr":"172.21.15.7:6821/859560576","heartbeat_back_addr":"172.21.15.7:6825/859560576","heartbeat_front_addr":"172.21.15.7:6823/859560576","state":["exists","up"]},{"osd":7,"uuid":"020ec859-3fc2-4da2-a0dc-10530a4a366c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6817","nonce":2578758635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6819","nonce":2578758635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6822","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6823","nonce":2578758635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6821","nonce":2578758635}]},"public_addr":"172.21.15.158:6817/2578758635","cluster_addr":"172.21.15.158:6819/2578758635","heartbeat_back_addr":"172.21.15.158:6823/2578758635","heartbeat_front_addr":"172.21.15.158:6821/2578758635","state":["exists","up"]},{"osd":8,"uuid":"4081496a-45ee-4b25-8f17-a23665521187","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6817","nonce":3020824843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6819","nonce":3020824843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6823","nonce":3020824843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6821","nonce":3020824843}]},"public_addr":"172.21.15.188:6817/3020824843","cluster_addr":"172.21.15.188:6819/3020824843","heartbeat_back_addr":"172.21.15.188:6823/3020824843","heartbeat_front_addr":"172.21.15.188:6821/3020824843","state":["exists","up"]},{"osd":9,"uuid":"723580ae-efe3-4d86-95fd-d7a4c0631425","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6827","nonce":1612464825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6829","nonce":1612464825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6833","nonce":1612464825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6831","nonce":1612464825}]},"public_addr":"172.21.15.7:6827/1612464825","cluster_addr":"172.21.15.7:6829/1612464825","heartbeat_back_addr":"172.21.15.7:6833/1612464825","heartbeat_front_addr":"172.21.15.7:6831/1612464825","state":["exists","up"]},{"osd":10,"uuid":"c2651548-fe46-4d2d-b911-278036412ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6825","nonce":1653166210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6827","nonce":1653166210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6831","nonce":1653166210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6829","nonce":1653166210}]},"public_addr":"172.21.15.158:6825/1653166210","cluster_addr":"172.21.15.158:6827/1653166210","heartbeat_back_addr":"172.21.15.158:6831/1653166210","heartbeat_front_addr":"172.21.15.158:6829/1653166210","state":["exists","up"]},{"osd":11,"uuid":"f7262bcb-4243-403a-a321-fc2bdf1d4d25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6825","nonce":1230196525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6827","nonce":1230196525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6831","nonce":1230196525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6829","nonce":1230196525}]},"public_addr":"172.21.15.188:6825/1230196525","cluster_addr":"172.21.15.188:6827/1230196525","heartbeat_back_addr":"172.21.15.188:6831/1230196525","heartbeat_front_addr":"172.21.15.188:6829/1230196525","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:46:26.803832+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:46:48.726253+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:47:09.744123+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:47:36.907588+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:47:57.696685+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:48:19.120806+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:48:45.313043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:49:06.686384+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:49:28.475078+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:49:54.984033+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:50:16.541259+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:50:38.221746+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/3892248290":"2023-07-21T02:44:53.458649+0000","172.21.15.7:0/2373058722":"2023-07-21T02:44:53.458649+0000","172.21.15.7:6801/2926310425":"2023-07-21T02:44:53.458649+0000","172.21.15.7:0/4222569264":"2023-07-21T02:44:25.804258+0000","172.21.15.7:0/1357065986":"2023-07-21T02:44:25.804258+0000","172.21.15.7:0/884368977":"2023-07-21T02:44:53.458649+0000","172.21.15.7:6801/1224472946":"2023-07-21T02:44:25.804258+0000","172.21.15.7:0/1459293748":"2023-07-21T02:44:25.804258+0000","172.21.15.7:6800/1224472946":"2023-07-21T02:44:25.804258+0000","172.21.15.7:6800/2926310425":"2023-07-21T02:44:53.458649+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T02:51:35.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.575+0000 7fabe29ff700 1 -- 172.21.15.7:0/3668575776 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fabc40a1220 msgr2=0x7fabc40a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:35.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.575+0000 7fabe29ff700 1 --2- 172.21.15.7:0/3668575776 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fabc40a1220 0x7fabc40a36e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fabd0009c60 tx=0x7fabd0009260 comp rx=0 tx=0).stop 2023-07-20T02:51:35.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.575+0000 7fabe29ff700 1 -- 172.21.15.7:0/3668575776 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabdc100770 msgr2=0x7fabdc1986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:35.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.575+0000 7fabe29ff700 1 --2- 172.21.15.7:0/3668575776 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabdc100770 0x7fabdc1986c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fabcc009fd0 tx=0x7fabcc00b560 comp rx=0 tx=0).stop 2023-07-20T02:51:35.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.575+0000 7fabe29ff700 1 -- 172.21.15.7:0/3668575776 shutdown_connections 2023-07-20T02:51:35.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.575+0000 7fabe29ff700 1 --2- 172.21.15.7:0/3668575776 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fabdc102300 0x7fabdc19fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:35.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.575+0000 7fabe29ff700 1 --2- 172.21.15.7:0/3668575776 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fabc40a1220 0x7fabc40a36e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:35.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.575+0000 7fabe29ff700 1 --2- 172.21.15.7:0/3668575776 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabdc100770 0x7fabdc1986c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:35.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.575+0000 7fabe29ff700 1 --2- 172.21.15.7:0/3668575776 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fabdc101960 0x7fabdc198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:35.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.575+0000 7fabe29ff700 1 -- 172.21.15.7:0/3668575776 >> 172.21.15.7:0/3668575776 conn(0x7fabdc0fbca0 msgr2=0x7fabdc107280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:35.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.576+0000 7fabe29ff700 1 -- 172.21.15.7:0/3668575776 shutdown_connections 2023-07-20T02:51:35.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.576+0000 7fabe29ff700 1 -- 172.21.15.7:0/3668575776 wait complete. 2023-07-20T02:51:35.625 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T02:51:35.640 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:35 smithi158 ceph-mon[128925]: osdmap e87: 12 total, 12 up, 12 in 2023-07-20T02:51:35.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:35 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/2367900216' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-07-20T02:51:35.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:35 smithi158 ceph-mon[128925]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T02:51:35.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:35 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/665950530' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:35.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:35 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/3674528279' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:51:35.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.815+0000 7fe7f3fff700 1 -- 172.21.15.7:0/3962413200 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7ec101960 msgr2=0x7fe7ec101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:35.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.815+0000 7fe7f3fff700 1 --2- 172.21.15.7:0/3962413200 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7ec101960 0x7fe7ec101dc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc0099c0 tx=0x7fe7dc009cd0 comp rx=0 tx=0).stop 2023-07-20T02:51:35.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.815+0000 7fe7f3fff700 1 -- 172.21.15.7:0/3962413200 shutdown_connections 2023-07-20T02:51:35.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.815+0000 7fe7f3fff700 1 --2- 172.21.15.7:0/3962413200 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7ec102300 0x7fe7ec108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:35.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.815+0000 7fe7f3fff700 1 --2- 172.21.15.7:0/3962413200 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe7ec100770 0x7fe7ec100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:35.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.815+0000 7fe7f3fff700 1 --2- 172.21.15.7:0/3962413200 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7ec101960 0x7fe7ec101dc0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:35.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.816+0000 7fe7f3fff700 1 -- 172.21.15.7:0/3962413200 >> 172.21.15.7:0/3962413200 conn(0x7fe7ec0fbca0 msgr2=0x7fe7ec0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:35.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.816+0000 7fe7f3fff700 1 -- 172.21.15.7:0/3962413200 shutdown_connections 2023-07-20T02:51:35.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.816+0000 7fe7f3fff700 1 -- 172.21.15.7:0/3962413200 wait complete. 2023-07-20T02:51:35.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.817+0000 7fe7f3fff700 1 Processor -- start 2023-07-20T02:51:35.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.817+0000 7fe7f3fff700 1 -- start start 2023-07-20T02:51:35.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.817+0000 7fe7f3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7ec100770 0x7fe7ec198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:35.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.817+0000 7fe7f3fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7ec101960 0x7fe7ec198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:35.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.817+0000 7fe7f3fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe7ec102300 0x7fe7ec19fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:35.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.817+0000 7fe7f3fff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ec1a01c0 con 0x7fe7ec100770 2023-07-20T02:51:35.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.817+0000 7fe7f3fff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ec1a0300 con 0x7fe7ec101960 2023-07-20T02:51:35.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.817+0000 7fe7f3fff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ec1a0490 con 0x7fe7ec102300 2023-07-20T02:51:35.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.817+0000 7fe7f1d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7ec100770 0x7fe7ec198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:35.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.817+0000 7fe7f1d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7ec100770 0x7fe7ec198640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51918/0 (socket says 172.21.15.7:51918) 2023-07-20T02:51:35.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.817+0000 7fe7f1d9b700 1 -- 172.21.15.7:0/739151192 learned_addr learned my addr 172.21.15.7:0/739151192 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:35.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.817+0000 7fe7f159a700 1 --2- 172.21.15.7:0/739151192 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7ec101960 0x7fe7ec198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:35.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.818+0000 7fe7f259c700 1 --2- 172.21.15.7:0/739151192 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe7ec102300 0x7fe7ec19fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:35.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.818+0000 7fe7f1d9b700 1 -- 172.21.15.7:0/739151192 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe7ec102300 msgr2=0x7fe7ec19fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:35.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.818+0000 7fe7f1d9b700 1 --2- 172.21.15.7:0/739151192 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe7ec102300 0x7fe7ec19fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:35.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.818+0000 7fe7f1d9b700 1 -- 172.21.15.7:0/739151192 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7ec101960 msgr2=0x7fe7ec198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:35.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.818+0000 7fe7f1d9b700 1 --2- 172.21.15.7:0/739151192 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7ec101960 0x7fe7ec198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:35.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.818+0000 7fe7f1d9b700 1 -- 172.21.15.7:0/739151192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7dc0095f0 con 0x7fe7ec100770 2023-07-20T02:51:35.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.818+0000 7fe7f159a700 1 --2- 172.21.15.7:0/739151192 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7ec101960 0x7fe7ec198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T02:51:35.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.818+0000 7fe7f1d9b700 1 --2- 172.21.15.7:0/739151192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7ec100770 0x7fe7ec198640 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe7e800e970 tx=0x7fe7e800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:35.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.818+0000 7fe7e2ffd700 1 -- 172.21.15.7:0/739151192 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8009790 con 0x7fe7ec100770 2023-07-20T02:51:35.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.818+0000 7fe7f3fff700 1 -- 172.21.15.7:0/739151192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7ec1a0770 con 0x7fe7ec100770 2023-07-20T02:51:35.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.818+0000 7fe7e2ffd700 1 -- 172.21.15.7:0/739151192 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe7e800eec0 con 0x7fe7ec100770 2023-07-20T02:51:35.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.818+0000 7fe7e2ffd700 1 -- 172.21.15.7:0/739151192 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8003cf0 con 0x7fe7ec100770 2023-07-20T02:51:35.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.818+0000 7fe7f3fff700 1 -- 172.21.15.7:0/739151192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7ec1a0cd0 con 0x7fe7ec100770 2023-07-20T02:51:35.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.819+0000 7fe7f3fff700 1 -- 172.21.15.7:0/739151192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7ec10bfe0 con 0x7fe7ec100770 2023-07-20T02:51:35.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.821+0000 7fe7e2ffd700 1 -- 172.21.15.7:0/739151192 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fe7e800ccd0 con 0x7fe7ec100770 2023-07-20T02:51:35.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.823+0000 7fe7e2ffd700 1 --2- 172.21.15.7:0/739151192 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fe7d80a11a0 0x7fe7d80a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:35.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.823+0000 7fe7e2ffd700 1 -- 172.21.15.7:0/739151192 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe7e8014070 con 0x7fe7ec100770 2023-07-20T02:51:35.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.823+0000 7fe7f159a700 1 --2- 172.21.15.7:0/739151192 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fe7d80a11a0 0x7fe7d80a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:35.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.824+0000 7fe7f159a700 1 --2- 172.21.15.7:0/739151192 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fe7d80a11a0 0x7fe7d80a3660 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc00b580 tx=0x7fe7dc01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:35.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:35.826+0000 7fe7e2ffd700 1 -- 172.21.15.7:0/739151192 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe7e80962b0 con 0x7fe7ec100770 2023-07-20T02:51:36.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.083+0000 7fe7f3fff700 1 -- 172.21.15.7:0/739151192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7ec1a1370 con 0x7fe7ec100770 2023-07-20T02:51:36.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.083+0000 7fe7e2ffd700 1 -- 172.21.15.7:0/739151192 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+10757 (secure 0 0 0) 0x7fe7e80998d0 con 0x7fe7ec100770 2023-07-20T02:51:36.097 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:51:36.098 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/190105946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":190105946},{"type":"v1","addr":"172.21.15.7:6837","nonce":190105946}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14556,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/3310846795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":3310846795},{"type":"v1","addr":"172.21.15.7:6839","nonce":3310846795}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14568,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3189534126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3189534126},{"type":"v1","addr":"172.21.15.7:6841","nonce":3189534126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24472,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/1435841351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":1435841351},{"type":"v1","addr":"172.21.15.188:6833","nonce":1435841351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6833/401861265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6832","nonce":401861265},{"type":"v1","addr":"172.21.15.158:6833","nonce":401861265}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/4064058311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":4064058311},{"type":"v1","addr":"172.21.15.7:6835","nonce":4064058311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6835/3931487788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6834","nonce":3931487788},{"type":"v1","addr":"172.21.15.158:6835","nonce":3931487788}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/857155289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":857155289},{"type":"v1","addr":"172.21.15.188:6835","nonce":857155289}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.158:6837/1032497949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6836","nonce":1032497949},{"type":"v1","addr":"172.21.15.158:6837","nonce":1032497949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/775436728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":775436728},{"type":"v1","addr":"172.21.15.188:6837","nonce":775436728}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24562,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6839/450074084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":450074084},{"type":"v1","addr":"172.21.15.188:6839","nonce":450074084}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-07-20T02:51:28.640448+0000","modified":"2023-07-20T02:51:34.171163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.158:6839/2552719389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6838","nonce":2552719389},{"type":"v1","addr":"172.21.15.158:6839","nonce":2552719389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-07-20T02:51:36.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.086+0000 7fe7f3fff700 1 -- 172.21.15.7:0/739151192 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fe7d80a11a0 msgr2=0x7fe7d80a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:36.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.086+0000 7fe7f3fff700 1 --2- 172.21.15.7:0/739151192 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fe7d80a11a0 0x7fe7d80a3660 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc00b580 tx=0x7fe7dc01a040 comp rx=0 tx=0).stop 2023-07-20T02:51:36.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.086+0000 7fe7f3fff700 1 -- 172.21.15.7:0/739151192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7ec100770 msgr2=0x7fe7ec198640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:36.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.086+0000 7fe7f3fff700 1 --2- 172.21.15.7:0/739151192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7ec100770 0x7fe7ec198640 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe7e800e970 tx=0x7fe7e800ec80 comp rx=0 tx=0).stop 2023-07-20T02:51:36.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.086+0000 7fe7f3fff700 1 -- 172.21.15.7:0/739151192 shutdown_connections 2023-07-20T02:51:36.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.086+0000 7fe7f3fff700 1 --2- 172.21.15.7:0/739151192 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe7ec102300 0x7fe7ec19fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.086+0000 7fe7f3fff700 1 --2- 172.21.15.7:0/739151192 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fe7d80a11a0 0x7fe7d80a3660 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.086+0000 7fe7f3fff700 1 --2- 172.21.15.7:0/739151192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7ec100770 0x7fe7ec198640 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.086+0000 7fe7f3fff700 1 --2- 172.21.15.7:0/739151192 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fe7ec101960 0x7fe7ec198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.086+0000 7fe7f3fff700 1 -- 172.21.15.7:0/739151192 >> 172.21.15.7:0/739151192 conn(0x7fe7ec0fbca0 msgr2=0x7fe7ec0fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:36.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.086+0000 7fe7f3fff700 1 -- 172.21.15.7:0/739151192 shutdown_connections 2023-07-20T02:51:36.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.086+0000 7fe7f3fff700 1 -- 172.21.15.7:0/739151192 wait complete. 2023-07-20T02:51:36.103 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 17 2023-07-20T02:51:36.120 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T02:51:36.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.312+0000 7f319f3ff700 1 -- 172.21.15.7:0/1772753732 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3198100770 msgr2=0x7f3198100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:36.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.312+0000 7f319f3ff700 1 --2- 172.21.15.7:0/1772753732 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3198100770 0x7f3198100b90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f31940099c0 tx=0x7f3194009cd0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.312+0000 7f319f3ff700 1 -- 172.21.15.7:0/1772753732 shutdown_connections 2023-07-20T02:51:36.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.312+0000 7f319f3ff700 1 --2- 172.21.15.7:0/1772753732 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3198102300 0x7f3198108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.312+0000 7f319f3ff700 1 --2- 172.21.15.7:0/1772753732 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3198100770 0x7f3198100b90 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.312+0000 7f319f3ff700 1 --2- 172.21.15.7:0/1772753732 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3198101960 0x7f3198101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.312+0000 7f319f3ff700 1 -- 172.21.15.7:0/1772753732 >> 172.21.15.7:0/1772753732 conn(0x7f31980fbca0 msgr2=0x7f31980fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:36.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.313+0000 7f319f3ff700 1 -- 172.21.15.7:0/1772753732 shutdown_connections 2023-07-20T02:51:36.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.313+0000 7f319f3ff700 1 -- 172.21.15.7:0/1772753732 wait complete. 2023-07-20T02:51:36.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.314+0000 7f319f3ff700 1 Processor -- start 2023-07-20T02:51:36.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.314+0000 7f319f3ff700 1 -- start start 2023-07-20T02:51:36.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.314+0000 7f319f3ff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3198100770 0x7f3198196440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:36.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.314+0000 7f319f3ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3198101960 0x7f3198196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:36.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.314+0000 7f319f3ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3198102300 0x7f319819d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:36.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.314+0000 7f319f3ff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319819dfc0 con 0x7f3198102300 2023-07-20T02:51:36.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.314+0000 7f319f3ff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319819e100 con 0x7f3198100770 2023-07-20T02:51:36.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.314+0000 7f319f3ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319819e290 con 0x7f3198101960 2023-07-20T02:51:36.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.314+0000 7f319d99c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3198102300 0x7f319819d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:36.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.314+0000 7f319d99c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3198102300 0x7f319819d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51932/0 (socket says 172.21.15.7:51932) 2023-07-20T02:51:36.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.314+0000 7f319d99c700 1 -- 172.21.15.7:0/3045656772 learned_addr learned my addr 172.21.15.7:0/3045656772 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:36.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.314+0000 7f319d19b700 1 --2- 172.21.15.7:0/3045656772 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3198100770 0x7f3198196440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:36.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.314+0000 7f319c99a700 1 --2- 172.21.15.7:0/3045656772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3198101960 0x7f3198196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:36.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.315+0000 7f319d99c700 1 -- 172.21.15.7:0/3045656772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3198101960 msgr2=0x7f3198196980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:36.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.315+0000 7f319d99c700 1 --2- 172.21.15.7:0/3045656772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3198101960 0x7f3198196980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.315+0000 7f319d99c700 1 -- 172.21.15.7:0/3045656772 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3198100770 msgr2=0x7f3198196440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:36.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.315+0000 7f319d99c700 1 --2- 172.21.15.7:0/3045656772 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3198100770 0x7f3198196440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.315+0000 7f319d99c700 1 -- 172.21.15.7:0/3045656772 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31940095f0 con 0x7f3198102300 2023-07-20T02:51:36.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.315+0000 7f319d99c700 1 --2- 172.21.15.7:0/3045656772 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3198102300 0x7f319819d9f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3190007ee0 tx=0x7f319000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:36.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.315+0000 7f318e7fc700 1 -- 172.21.15.7:0/3045656772 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3190017070 con 0x7f3198102300 2023-07-20T02:51:36.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.315+0000 7f318e7fc700 1 -- 172.21.15.7:0/3045656772 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f319000f040 con 0x7f3198102300 2023-07-20T02:51:36.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.315+0000 7f318e7fc700 1 -- 172.21.15.7:0/3045656772 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3190005580 con 0x7f3198102300 2023-07-20T02:51:36.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.315+0000 7f319f3ff700 1 -- 172.21.15.7:0/3045656772 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f319819e570 con 0x7f3198102300 2023-07-20T02:51:36.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.315+0000 7f319f3ff700 1 -- 172.21.15.7:0/3045656772 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f319819ea30 con 0x7f3198102300 2023-07-20T02:51:36.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.316+0000 7f319f3ff700 1 -- 172.21.15.7:0/3045656772 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3198107cb0 con 0x7f3198102300 2023-07-20T02:51:36.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.319+0000 7f318e7fc700 1 -- 172.21.15.7:0/3045656772 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f319000db20 con 0x7f3198102300 2023-07-20T02:51:36.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.321+0000 7f318e7fc700 1 --2- 172.21.15.7:0/3045656772 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f31840a1220 0x7f31840a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:36.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.321+0000 7f318e7fc700 1 -- 172.21.15.7:0/3045656772 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f31900cc260 con 0x7f3198102300 2023-07-20T02:51:36.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.321+0000 7f319d19b700 1 --2- 172.21.15.7:0/3045656772 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f31840a1220 0x7f31840a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:36.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.322+0000 7f319d19b700 1 --2- 172.21.15.7:0/3045656772 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f31840a1220 0x7f31840a36e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3194009990 tx=0x7f3194005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:36.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.325+0000 7f318e7fc700 1 -- 172.21.15.7:0/3045656772 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3190095760 con 0x7f3198102300 2023-07-20T02:51:36.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:36 smithi007 ceph-mon[126353]: pgmap v278: 129 pgs: 113 active+clean, 16 unknown; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 3 op/s 2023-07-20T02:51:36.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:36 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/3668575776' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:51:36.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:36 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/739151192' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:36.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:36 smithi188 ceph-mon[129553]: pgmap v278: 129 pgs: 113 active+clean, 16 unknown; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 3 op/s 2023-07-20T02:51:36.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:36 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/3668575776' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:51:36.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:36 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/739151192' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:36.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.534+0000 7f319f3ff700 1 -- 172.21.15.7:0/3045656772 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f319802c970 con 0x7f3198102300 2023-07-20T02:51:36.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.536+0000 7f318e7fc700 1 -- 172.21.15.7:0/3045656772 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+23927 (secure 0 0 0) 0x7f3190098d80 con 0x7f3198102300 2023-07-20T02:51:36.550 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:51:36.550 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":87,"fsid":"1b04643c-26a7-11ee-9b34-001a4aab830c","created":"2023-07-20T02:43:56.544719+0000","modified":"2023-07-20T02:51:33.697865+0000","last_up_change":"2023-07-20T02:50:40.126926+0000","last_in_change":"2023-07-20T02:50:21.875020+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T02:47:12.055539+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-20T02:51:26.617961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-20T02:51:28.081016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-20T02:51:31.658088+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ad6c6339-6e23-4c21-9fde-2dbecdafdb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6803","nonce":3322992091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6805","nonce":3322992091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6809","nonce":3322992091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6807","nonce":3322992091}]},"public_addr":"172.21.15.7:6803/3322992091","cluster_addr":"172.21.15.7:6805/3322992091","heartbeat_back_addr":"172.21.15.7:6809/3322992091","heartbeat_front_addr":"172.21.15.7:6807/3322992091","state":["exists","up"]},{"osd":1,"uuid":"cad207b3-a1ea-4f07-bd94-9b11fd573a69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6801","nonce":2796771612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6803","nonce":2796771612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6807","nonce":2796771612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6805","nonce":2796771612}]},"public_addr":"172.21.15.158:6801/2796771612","cluster_addr":"172.21.15.158:6803/2796771612","heartbeat_back_addr":"172.21.15.158:6807/2796771612","heartbeat_front_addr":"172.21.15.158:6805/2796771612","state":["exists","up"]},{"osd":2,"uuid":"2c777fe6-d003-49d0-af37-f63fca37ff06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6801","nonce":2928307258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6803","nonce":2928307258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6807","nonce":2928307258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6805","nonce":2928307258}]},"public_addr":"172.21.15.188:6801/2928307258","cluster_addr":"172.21.15.188:6803/2928307258","heartbeat_back_addr":"172.21.15.188:6807/2928307258","heartbeat_front_addr":"172.21.15.188:6805/2928307258","state":["exists","up"]},{"osd":3,"uuid":"59e29965-018f-4eb6-adbd-1a9e1df84245","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6811","nonce":2192953827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6813","nonce":2192953827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6817","nonce":2192953827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6815","nonce":2192953827}]},"public_addr":"172.21.15.7:6811/2192953827","cluster_addr":"172.21.15.7:6813/2192953827","heartbeat_back_addr":"172.21.15.7:6817/2192953827","heartbeat_front_addr":"172.21.15.7:6815/2192953827","state":["exists","up"]},{"osd":4,"uuid":"e2453384-3b09-4f2a-bea7-172d8fc466d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6809","nonce":3330591132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6810","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6811","nonce":3330591132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6814","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6815","nonce":3330591132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6813","nonce":3330591132}]},"public_addr":"172.21.15.158:6809/3330591132","cluster_addr":"172.21.15.158:6811/3330591132","heartbeat_back_addr":"172.21.15.158:6815/3330591132","heartbeat_front_addr":"172.21.15.158:6813/3330591132","state":["exists","up"]},{"osd":5,"uuid":"c3c04c13-0ce5-440d-9e62-0e301ad9f771","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6809","nonce":3142012805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6811","nonce":3142012805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6815","nonce":3142012805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6813","nonce":3142012805}]},"public_addr":"172.21.15.188:6809/3142012805","cluster_addr":"172.21.15.188:6811/3142012805","heartbeat_back_addr":"172.21.15.188:6815/3142012805","heartbeat_front_addr":"172.21.15.188:6813/3142012805","state":["exists","up"]},{"osd":6,"uuid":"82ef2d33-1045-41f1-bf69-b03404bcace8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6819","nonce":859560576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6821","nonce":859560576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6825","nonce":859560576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6823","nonce":859560576}]},"public_addr":"172.21.15.7:6819/859560576","cluster_addr":"172.21.15.7:6821/859560576","heartbeat_back_addr":"172.21.15.7:6825/859560576","heartbeat_front_addr":"172.21.15.7:6823/859560576","state":["exists","up"]},{"osd":7,"uuid":"020ec859-3fc2-4da2-a0dc-10530a4a366c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6817","nonce":2578758635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6819","nonce":2578758635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6822","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6823","nonce":2578758635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6821","nonce":2578758635}]},"public_addr":"172.21.15.158:6817/2578758635","cluster_addr":"172.21.15.158:6819/2578758635","heartbeat_back_addr":"172.21.15.158:6823/2578758635","heartbeat_front_addr":"172.21.15.158:6821/2578758635","state":["exists","up"]},{"osd":8,"uuid":"4081496a-45ee-4b25-8f17-a23665521187","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6817","nonce":3020824843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6819","nonce":3020824843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6823","nonce":3020824843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6821","nonce":3020824843}]},"public_addr":"172.21.15.188:6817/3020824843","cluster_addr":"172.21.15.188:6819/3020824843","heartbeat_back_addr":"172.21.15.188:6823/3020824843","heartbeat_front_addr":"172.21.15.188:6821/3020824843","state":["exists","up"]},{"osd":9,"uuid":"723580ae-efe3-4d86-95fd-d7a4c0631425","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6827","nonce":1612464825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6829","nonce":1612464825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6833","nonce":1612464825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6831","nonce":1612464825}]},"public_addr":"172.21.15.7:6827/1612464825","cluster_addr":"172.21.15.7:6829/1612464825","heartbeat_back_addr":"172.21.15.7:6833/1612464825","heartbeat_front_addr":"172.21.15.7:6831/1612464825","state":["exists","up"]},{"osd":10,"uuid":"c2651548-fe46-4d2d-b911-278036412ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6825","nonce":1653166210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6827","nonce":1653166210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6831","nonce":1653166210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6829","nonce":1653166210}]},"public_addr":"172.21.15.158:6825/1653166210","cluster_addr":"172.21.15.158:6827/1653166210","heartbeat_back_addr":"172.21.15.158:6831/1653166210","heartbeat_front_addr":"172.21.15.158:6829/1653166210","state":["exists","up"]},{"osd":11,"uuid":"f7262bcb-4243-403a-a321-fc2bdf1d4d25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6825","nonce":1230196525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6827","nonce":1230196525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6831","nonce":1230196525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6829","nonce":1230196525}]},"public_addr":"172.21.15.188:6825/1230196525","cluster_addr":"172.21.15.188:6827/1230196525","heartbeat_back_addr":"172.21.15.188:6831/1230196525","heartbeat_front_addr":"172.21.15.188:6829/1230196525","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:46:26.803832+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:46:48.726253+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:47:09.744123+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:47:36.907588+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:47:57.696685+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:48:19.120806+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:48:45.313043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:49:06.686384+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:49:28.475078+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:49:54.984033+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:50:16.541259+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:50:38.221746+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/3892248290":"2023-07-21T02:44:53.458649+0000","172.21.15.7:0/2373058722":"2023-07-21T02:44:53.458649+0000","172.21.15.7:6801/2926310425":"2023-07-21T02:44:53.458649+0000","172.21.15.7:0/4222569264":"2023-07-21T02:44:25.804258+0000","172.21.15.7:0/1357065986":"2023-07-21T02:44:25.804258+0000","172.21.15.7:0/884368977":"2023-07-21T02:44:53.458649+0000","172.21.15.7:6801/1224472946":"2023-07-21T02:44:25.804258+0000","172.21.15.7:0/1459293748":"2023-07-21T02:44:25.804258+0000","172.21.15.7:6800/1224472946":"2023-07-21T02:44:25.804258+0000","172.21.15.7:6800/2926310425":"2023-07-21T02:44:53.458649+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T02:51:36.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.538+0000 7f319f3ff700 1 -- 172.21.15.7:0/3045656772 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f31840a1220 msgr2=0x7f31840a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:36.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.538+0000 7f319f3ff700 1 --2- 172.21.15.7:0/3045656772 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f31840a1220 0x7f31840a36e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3194009990 tx=0x7f3194005fb0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.538+0000 7f319f3ff700 1 -- 172.21.15.7:0/3045656772 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3198102300 msgr2=0x7f319819d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:36.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.538+0000 7f319f3ff700 1 --2- 172.21.15.7:0/3045656772 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3198102300 0x7f319819d9f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3190007ee0 tx=0x7f319000d3b0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.538+0000 7f319f3ff700 1 -- 172.21.15.7:0/3045656772 shutdown_connections 2023-07-20T02:51:36.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.538+0000 7f319f3ff700 1 --2- 172.21.15.7:0/3045656772 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f31840a1220 0x7f31840a36e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.538+0000 7f319f3ff700 1 --2- 172.21.15.7:0/3045656772 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3198102300 0x7f319819d9f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.538+0000 7f319f3ff700 1 --2- 172.21.15.7:0/3045656772 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3198100770 0x7f3198196440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.538+0000 7f319f3ff700 1 --2- 172.21.15.7:0/3045656772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3198101960 0x7f3198196980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.538+0000 7f319f3ff700 1 -- 172.21.15.7:0/3045656772 >> 172.21.15.7:0/3045656772 conn(0x7f31980fbca0 msgr2=0x7f3198104df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:36.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.539+0000 7f319f3ff700 1 -- 172.21.15.7:0/3045656772 shutdown_connections 2023-07-20T02:51:36.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.539+0000 7f319f3ff700 1 -- 172.21.15.7:0/3045656772 wait complete. 2023-07-20T02:51:36.572 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-07-20T02:51:28.640448+0000', 'modified': '2023-07-20T02:51:34.171163+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24572}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24572': {'gid': 24572, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.158:6839/2552719389', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.158:6838', 'nonce': 2552719389}, {'type': 'v1', 'addr': '172.21.15.158:6839', 'nonce': 2552719389}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2023-07-20T02:51:36.572 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-07-20T02:51:36.573 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-07-20T02:51:36.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:36 smithi158 ceph-mon[128925]: pgmap v278: 129 pgs: 113 active+clean, 16 unknown; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 3 op/s 2023-07-20T02:51:36.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:36 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/3668575776' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:51:36.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:36 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/739151192' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T02:51:36.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.764+0000 7f41df7ff700 1 -- 172.21.15.7:0/789436836 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41d8101950 msgr2=0x7f41d8101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:36.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.764+0000 7f41df7ff700 1 --2- 172.21.15.7:0/789436836 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41d8101950 0x7f41d8101db0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f41c80099c0 tx=0x7f41c8009cd0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.765+0000 7f41df7ff700 1 -- 172.21.15.7:0/789436836 shutdown_connections 2023-07-20T02:51:36.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.765+0000 7f41df7ff700 1 --2- 172.21.15.7:0/789436836 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41d81022f0 0x7f41d8108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.765+0000 7f41df7ff700 1 --2- 172.21.15.7:0/789436836 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41d8100760 0x7f41d8100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.765+0000 7f41df7ff700 1 --2- 172.21.15.7:0/789436836 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41d8101950 0x7f41d8101db0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.765+0000 7f41df7ff700 1 -- 172.21.15.7:0/789436836 >> 172.21.15.7:0/789436836 conn(0x7f41d80fbcb0 msgr2=0x7f41d80fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:36.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.765+0000 7f41df7ff700 1 -- 172.21.15.7:0/789436836 shutdown_connections 2023-07-20T02:51:36.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.765+0000 7f41df7ff700 1 -- 172.21.15.7:0/789436836 wait complete. 2023-07-20T02:51:36.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.766+0000 7f41df7ff700 1 Processor -- start 2023-07-20T02:51:36.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.766+0000 7f41df7ff700 1 -- start start 2023-07-20T02:51:36.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.766+0000 7f41df7ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41d8100760 0x7f41d8196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:36.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.766+0000 7f41df7ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41d8101950 0x7f41d81969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:36.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.766+0000 7f41df7ff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41d81022f0 0x7f41d819da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:36.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.766+0000 7f41df7ff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d819dff0 con 0x7f41d8100760 2023-07-20T02:51:36.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.766+0000 7f41df7ff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d819e130 con 0x7f41d81022f0 2023-07-20T02:51:36.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.766+0000 7f41df7ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d819e2c0 con 0x7f41d8101950 2023-07-20T02:51:36.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.766+0000 7f41dd59b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41d8100760 0x7f41d8196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:36.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.767+0000 7f41dd59b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41d8100760 0x7f41d8196470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51944/0 (socket says 172.21.15.7:51944) 2023-07-20T02:51:36.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.767+0000 7f41ddd9c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41d81022f0 0x7f41d819da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:36.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.767+0000 7f41dd59b700 1 -- 172.21.15.7:0/4285386125 learned_addr learned my addr 172.21.15.7:0/4285386125 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:36.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.767+0000 7f41dcd9a700 1 --2- 172.21.15.7:0/4285386125 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41d8101950 0x7f41d81969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:36.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.767+0000 7f41dd59b700 1 -- 172.21.15.7:0/4285386125 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41d8101950 msgr2=0x7f41d81969b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:36.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.767+0000 7f41dd59b700 1 --2- 172.21.15.7:0/4285386125 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41d8101950 0x7f41d81969b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.767+0000 7f41dd59b700 1 -- 172.21.15.7:0/4285386125 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41d81022f0 msgr2=0x7f41d819da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:36.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.767+0000 7f41dd59b700 1 --2- 172.21.15.7:0/4285386125 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41d81022f0 0x7f41d819da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.767+0000 7f41dd59b700 1 -- 172.21.15.7:0/4285386125 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c80095f0 con 0x7f41d8100760 2023-07-20T02:51:36.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.767+0000 7f41dd59b700 1 --2- 172.21.15.7:0/4285386125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41d8100760 0x7f41d8196470 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f41d400e970 tx=0x7f41d400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:36.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.767+0000 7f41ce7fc700 1 -- 172.21.15.7:0/4285386125 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d4009790 con 0x7f41d8100760 2023-07-20T02:51:36.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.767+0000 7f41ce7fc700 1 -- 172.21.15.7:0/4285386125 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f41d400c7e0 con 0x7f41d8100760 2023-07-20T02:51:36.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.767+0000 7f41ce7fc700 1 -- 172.21.15.7:0/4285386125 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d4003c70 con 0x7f41d8100760 2023-07-20T02:51:36.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.767+0000 7f41df7ff700 1 -- 172.21.15.7:0/4285386125 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41d819e5a0 con 0x7f41d8100760 2023-07-20T02:51:36.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.768+0000 7f41df7ff700 1 -- 172.21.15.7:0/4285386125 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41d819ea60 con 0x7f41d8100760 2023-07-20T02:51:36.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.771+0000 7f41ce7fc700 1 -- 172.21.15.7:0/4285386125 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f41d400cc50 con 0x7f41d8100760 2023-07-20T02:51:36.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.772+0000 7f41f3035b80 1 Processor -- start 2023-07-20T02:51:36.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.772+0000 7f41f3035b80 1 -- start start 2023-07-20T02:51:36.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.772+0000 7f41ce7fc700 1 --2- 172.21.15.7:0/4285386125 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f41c40a11d0 0x7f41c40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:36.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.772+0000 7f41ce7fc700 1 -- 172.21.15.7:0/4285386125 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f41d4014070 con 0x7f41d8100760 2023-07-20T02:51:36.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.772+0000 7f41f3035b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631a4cf5540 0x5631a4cf5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:36.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.772+0000 7f41f3035b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5631a4cf5ea0 0x5631a4cf6320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:36.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.772+0000 7f41f3035b80 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631a4cf6900 0x5631a4cf6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:36.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.772+0000 7f41f3035b80 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631a4c70d70 con 0x5631a4cf6900 2023-07-20T02:51:36.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.772+0000 7f41f3035b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631a4c4d540 con 0x5631a4cf5ea0 2023-07-20T02:51:36.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.773+0000 7f41f3035b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631a4c4ae60 con 0x5631a4cf5540 2023-07-20T02:51:36.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.773+0000 7f41ddd9c700 1 --2- 172.21.15.7:0/4285386125 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f41c40a11d0 0x7f41c40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:36.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.773+0000 7f41dcd9a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5631a4cf5ea0 0x5631a4cf6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:36.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.773+0000 7f41dcd9a700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5631a4cf5ea0 0x5631a4cf6320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.7:60644/0 (socket says 172.21.15.7:60644) 2023-07-20T02:51:36.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.773+0000 7f41dcd9a700 1 -- 172.21.15.7:0/720492343 learned_addr learned my addr 172.21.15.7:0/720492343 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:36.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.773+0000 7f41dd59b700 1 --2- 172.21.15.7:0/720492343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631a4cf5540 0x5631a4cf5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:36.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.773+0000 7f41ddd9c700 1 --2- 172.21.15.7:0/720492343 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631a4cf6900 0x5631a4cf6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:36.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.773+0000 7f41dcd9a700 1 -- 172.21.15.7:0/720492343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631a4cf5540 msgr2=0x5631a4cf5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-20T02:51:36.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.773+0000 7f41dcd9a700 1 --2- 172.21.15.7:0/720492343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631a4cf5540 0x5631a4cf5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.773+0000 7f41dcd9a700 1 -- 172.21.15.7:0/720492343 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631a4cf6900 msgr2=0x5631a4cf6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-20T02:51:36.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.773+0000 7f41dcd9a700 1 --2- 172.21.15.7:0/720492343 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631a4cf6900 0x5631a4cf6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.773+0000 7f41dcd9a700 1 -- 172.21.15.7:0/720492343 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5631a4ac75e0 con 0x5631a4cf5ea0 2023-07-20T02:51:36.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.773+0000 7f41ddd9c700 1 --2- 172.21.15.7:0/4285386125 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f41c40a11d0 0x7f41c40a3690 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f41d000f870 tx=0x7f41d000fc30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:36.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.774+0000 7f41dcd9a700 1 --2- 172.21.15.7:0/720492343 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5631a4cf5ea0 0x5631a4cf6320 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f41c80099c0 tx=0x7f41c8005ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ce96f966f94f65f server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:36.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.774+0000 7f41c37fe700 1 -- 172.21.15.7:0/720492343 <== mon.2 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c800bad0 con 0x5631a4cf5ea0 2023-07-20T02:51:36.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.774+0000 7f41c37fe700 1 -- 172.21.15.7:0/720492343 <== mon.2 v2:172.21.15.158:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f41c801e040 con 0x5631a4cf5ea0 2023-07-20T02:51:36.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.774+0000 7f41c37fe700 1 -- 172.21.15.7:0/720492343 <== mon.2 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c8003ac0 con 0x5631a4cf5ea0 2023-07-20T02:51:36.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.775+0000 7f41f3035b80 1 -- 172.21.15.7:0/720492343 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5631a4cf5ea0 msgr2=0x5631a4cf6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:36.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.775+0000 7f41f3035b80 1 --2- 172.21.15.7:0/720492343 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5631a4cf5ea0 0x5631a4cf6320 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f41c80099c0 tx=0x7f41c8005ee0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.775+0000 7f41f3035b80 1 -- 172.21.15.7:0/720492343 shutdown_connections 2023-07-20T02:51:36.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.775+0000 7f41f3035b80 1 --2- 172.21.15.7:0/720492343 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631a4cf6900 0x5631a4cf6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.775+0000 7f41f3035b80 1 --2- 172.21.15.7:0/720492343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631a4cf5540 0x5631a4cf5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.775+0000 7f41f3035b80 1 --2- 172.21.15.7:0/720492343 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5631a4cf5ea0 0x5631a4cf6320 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.775+0000 7f41f3035b80 1 -- 172.21.15.7:0/720492343 >> 172.21.15.7:0/720492343 conn(0x5631a4c52c80 msgr2=0x5631a4cf0230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:36.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.775+0000 7f41f3035b80 1 -- 172.21.15.7:0/720492343 shutdown_connections 2023-07-20T02:51:36.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.775+0000 7f41f3035b80 1 -- 172.21.15.7:0/720492343 wait complete. 2023-07-20T02:51:36.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.775+0000 7f41f3035b80 1 -- start start 2023-07-20T02:51:36.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.776+0000 7f41f3035b80 1 Processor -- start 2023-07-20T02:51:36.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.776+0000 7f41f3035b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631a4cf5540 0x5631a4cfeec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:36.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.777+0000 7f41f3035b80 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631a4cf5ea0 0x5631a4cff400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:36.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.777+0000 7f41f3035b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5631a4cf6900 0x5631a4d06440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:36.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.777+0000 7f41f3035b80 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631a4c46750 con 0x5631a4cf5ea0 2023-07-20T02:51:36.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.777+0000 7f41f3035b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631a4be5a70 con 0x5631a4cf6900 2023-07-20T02:51:36.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.777+0000 7f41f3035b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631a4be5770 con 0x5631a4cf5540 2023-07-20T02:51:36.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.777+0000 7f41f3035b80 10 client.0 _ll_register_callbacks cb 0x5631a4be3630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-20T02:51:36.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.777+0000 7f41dcd9a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631a4cf5ea0 0x5631a4cff400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:36.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.777+0000 7f41dcd9a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631a4cf5ea0 0x5631a4cff400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51960/0 (socket says 172.21.15.7:51960) 2023-07-20T02:51:36.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.777+0000 7f41dcd9a700 1 -- 172.21.15.7:0/3087947190 learned_addr learned my addr 172.21.15.7:0/3087947190 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:36.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.777+0000 7f41ddd9c700 1 --2- 172.21.15.7:0/3087947190 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5631a4cf6900 0x5631a4d06440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:36.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.777+0000 7f41dd59b700 1 --2- 172.21.15.7:0/3087947190 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631a4cf5540 0x5631a4cfeec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:36.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.777+0000 7f41dcd9a700 1 -- 172.21.15.7:0/3087947190 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631a4cf5540 msgr2=0x5631a4cfeec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-20T02:51:36.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.777+0000 7f41dcd9a700 1 --2- 172.21.15.7:0/3087947190 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631a4cf5540 0x5631a4cfeec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.777+0000 7f41dcd9a700 1 -- 172.21.15.7:0/3087947190 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5631a4cf6900 msgr2=0x5631a4d06440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-20T02:51:36.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.777+0000 7f41dcd9a700 1 --2- 172.21.15.7:0/3087947190 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5631a4cf6900 0x5631a4d06440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.777+0000 7f41dcd9a700 1 -- 172.21.15.7:0/3087947190 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41d81094f0 con 0x5631a4cf5ea0 2023-07-20T02:51:36.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.778+0000 7f41dcd9a700 1 --2- 172.21.15.7:0/3087947190 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631a4cf5ea0 0x5631a4cff400 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f41c800b580 tx=0x7f41c8005940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98c8a97558ba84ba server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:36.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.778+0000 7f41c0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2023-07-20T02:51:36.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.778+0000 7f41c0ff9700 1 -- 172.21.15.7:0/3087947190 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c8003840 con 0x5631a4cf5ea0 2023-07-20T02:51:36.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.778+0000 7f41f3035b80 1 -- 172.21.15.7:0/3087947190 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5631a4ce8420 con 0x5631a4cf5ea0 2023-07-20T02:51:36.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.778+0000 7f41c0ff9700 1 -- 172.21.15.7:0/3087947190 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f41c80039a0 con 0x5631a4cf5ea0 2023-07-20T02:51:36.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.778+0000 7f41c0ff9700 1 -- 172.21.15.7:0/3087947190 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f41c8003b60 con 0x5631a4cf5ea0 2023-07-20T02:51:36.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.778+0000 7f41f3035b80 10 client.14685 fetch_fsmap learned FSMap version 17 2023-07-20T02:51:36.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.778+0000 7f41f3035b80 1 -- 172.21.15.7:0/3087947190 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x5631a4a73d90 con 0x5631a4cf5ea0 2023-07-20T02:51:36.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.778+0000 7f41c0ff9700 1 -- 172.21.15.7:0/3087947190 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5631a4ce8420 con 0x5631a4cf5ea0 2023-07-20T02:51:36.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.779+0000 7f41c0ff9700 1 -- 172.21.15.7:0/3087947190 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 17) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f41c802bcf0 con 0x5631a4cf5ea0 2023-07-20T02:51:36.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.779+0000 7f41f3035b80 10 client.14685 fetch_fsmap finished waiting for FSMap version 17 2023-07-20T02:51:36.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.779+0000 7f41f3035b80 10 client.14685 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-20T02:51:36.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.779+0000 7f41f3035b80 20 client.14685 populate_metadata read hostname 'smithi007' 2023-07-20T02:51:36.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.779+0000 7f41f3035b80 1 --2- 172.21.15.7:0/3087947190 >> [v2:172.21.15.158:6838/2552719389,v1:172.21.15.158:6839/2552719389] conn(0x5631a4d09cb0 0x5631a4d09370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:36.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.779+0000 7f41f3035b80 4 client.14685 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-20T02:51:36.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.779+0000 7f41f3035b80 1 -- 172.21.15.7:0/3087947190 --> [v2:172.21.15.158:6838/2552719389,v1:172.21.15.158:6839/2552719389] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5631a4cc4980 con 0x5631a4d09cb0 2023-07-20T02:51:36.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.779+0000 7f41dd59b700 1 --2- 172.21.15.7:0/3087947190 >> [v2:172.21.15.158:6838/2552719389,v1:172.21.15.158:6839/2552719389] conn(0x5631a4d09cb0 0x5631a4d09370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:36.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.780+0000 7f41dd59b700 1 --2- 172.21.15.7:0/3087947190 >> [v2:172.21.15.158:6838/2552719389,v1:172.21.15.158:6839/2552719389] conn(0x5631a4d09cb0 0x5631a4d09370 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7809150d5867c1c6 server_cookie=d9ebf3dbbc01819e in_seq=0 out_seq=0 2023-07-20T02:51:36.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.780+0000 7f41c0ff9700 10 client.14685 ms_handle_connect on v2:172.21.15.158:6838/2552719389 2023-07-20T02:51:36.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.780+0000 7f41c0ff9700 1 -- 172.21.15.7:0/3087947190 <== mds.0 v2:172.21.15.158:6838/2552719389 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x5631a4cc4980 con 0x5631a4d09cb0 2023-07-20T02:51:36.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.780+0000 7f41c0ff9700 10 client.14685 handle_command_reply: tid=0 2023-07-20T02:51:36.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.781+0000 7f41f3035b80 1 client.14685 shutdown 2023-07-20T02:51:36.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.781+0000 7f41f3035b80 1 -- 172.21.15.7:0/3087947190 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631a4cf5ea0 msgr2=0x5631a4cff400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:36.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.781+0000 7f41f3035b80 1 --2- 172.21.15.7:0/3087947190 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631a4cf5ea0 0x5631a4cff400 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f41c800b580 tx=0x7f41c8005940 comp rx=0 tx=0).stop 2023-07-20T02:51:36.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.781+0000 7f41f3035b80 1 -- 172.21.15.7:0/3087947190 shutdown_connections 2023-07-20T02:51:36.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.781+0000 7f41f3035b80 1 --2- 172.21.15.7:0/3087947190 >> [v2:172.21.15.158:6838/2552719389,v1:172.21.15.158:6839/2552719389] conn(0x5631a4d09cb0 0x5631a4d09370 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.781+0000 7f41f3035b80 1 --2- 172.21.15.7:0/3087947190 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5631a4cf6900 0x5631a4d06440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.781+0000 7f41c0ff9700 0 client.14685 ms_handle_reset on v2:172.21.15.158:6838/2552719389 2023-07-20T02:51:36.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.781+0000 7f41f3035b80 1 --2- 172.21.15.7:0/3087947190 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631a4cf5540 0x5631a4cfeec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.781+0000 7f41f3035b80 1 --2- 172.21.15.7:0/3087947190 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631a4cf5ea0 0x5631a4cff400 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.781+0000 7f41f3035b80 1 -- 172.21.15.7:0/3087947190 >> 172.21.15.7:0/3087947190 conn(0x5631a4c52c80 msgr2=0x5631a4cef930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:36.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.781+0000 7f41f3035b80 1 -- 172.21.15.7:0/3087947190 shutdown_connections 2023-07-20T02:51:36.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.781+0000 7f41f3035b80 1 -- 172.21.15.7:0/3087947190 wait complete. 2023-07-20T02:51:36.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.781+0000 7f41f3035b80 20 client.14685 trim_cache size 0 max 16384 2023-07-20T02:51:36.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.799+0000 7f41f3035b80 1 Processor -- start 2023-07-20T02:51:36.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.799+0000 7f41f3035b80 1 -- start start 2023-07-20T02:51:36.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.800+0000 7f41f3035b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5631a4cf5540 0x5631a4d09370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:36.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.800+0000 7f41f3035b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631a4cf5ea0 0x5631a4d063d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:36.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.800+0000 7f41f3035b80 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631a4cf6900 0x5631a4cf9780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:36.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.800+0000 7f41f3035b80 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631a4ac98d0 con 0x5631a4cf6900 2023-07-20T02:51:36.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.800+0000 7f41f3035b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631a4cfd610 con 0x5631a4cf5540 2023-07-20T02:51:36.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.800+0000 7f41f3035b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631a4bef470 con 0x5631a4cf5ea0 2023-07-20T02:51:36.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.800+0000 7f41ddd9c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631a4cf6900 0x5631a4cf9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:36.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.800+0000 7f41ddd9c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631a4cf6900 0x5631a4cf9780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51976/0 (socket says 172.21.15.7:51976) 2023-07-20T02:51:36.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.800+0000 7f41dcd9a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631a4cf5ea0 0x5631a4d063d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:36.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.800+0000 7f41ddd9c700 1 -- 172.21.15.7:0/1857883161 learned_addr learned my addr 172.21.15.7:0/1857883161 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:36.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.800+0000 7f41dd59b700 1 --2- 172.21.15.7:0/1857883161 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5631a4cf5540 0x5631a4d09370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:36.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.800+0000 7f41ddd9c700 1 -- 172.21.15.7:0/1857883161 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631a4cf5ea0 msgr2=0x5631a4d063d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-20T02:51:36.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.800+0000 7f41ddd9c700 1 --2- 172.21.15.7:0/1857883161 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631a4cf5ea0 0x5631a4d063d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.800+0000 7f41ddd9c700 1 -- 172.21.15.7:0/1857883161 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5631a4cf5540 msgr2=0x5631a4d09370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-20T02:51:36.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.800+0000 7f41ddd9c700 1 --2- 172.21.15.7:0/1857883161 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5631a4cf5540 0x5631a4d09370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.800+0000 7f41ddd9c700 1 -- 172.21.15.7:0/1857883161 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5631a4a5cc10 con 0x5631a4cf6900 2023-07-20T02:51:36.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.800+0000 7f41dd59b700 1 --2- 172.21.15.7:0/1857883161 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5631a4cf5540 0x5631a4d09370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T02:51:36.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.800+0000 7f41dcd9a700 1 --2- 172.21.15.7:0/1857883161 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631a4cf5ea0 0x5631a4d063d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T02:51:36.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.801+0000 7f41ddd9c700 1 --2- 172.21.15.7:0/1857883161 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631a4cf6900 0x5631a4cf9780 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f41d00092f0 tx=0x7f41d001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83457164fb1bc864 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:36.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.801+0000 7f41c3fff700 1 -- 172.21.15.7:0/1857883161 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d000e040 con 0x5631a4cf6900 2023-07-20T02:51:36.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.801+0000 7f41c3fff700 1 -- 172.21.15.7:0/1857883161 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f41d000f040 con 0x5631a4cf6900 2023-07-20T02:51:36.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.801+0000 7f41c3fff700 1 -- 172.21.15.7:0/1857883161 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d001a6a0 con 0x5631a4cf6900 2023-07-20T02:51:36.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.801+0000 7f41f3035b80 1 -- 172.21.15.7:0/1857883161 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631a4cf6900 msgr2=0x5631a4cf9780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:36.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.801+0000 7f41f3035b80 1 --2- 172.21.15.7:0/1857883161 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631a4cf6900 0x5631a4cf9780 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f41d00092f0 tx=0x7f41d001f000 comp rx=0 tx=0).stop 2023-07-20T02:51:36.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.802+0000 7f41f3035b80 1 -- 172.21.15.7:0/1857883161 shutdown_connections 2023-07-20T02:51:36.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.802+0000 7f41f3035b80 1 --2- 172.21.15.7:0/1857883161 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631a4cf6900 0x5631a4cf9780 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.802+0000 7f41f3035b80 1 --2- 172.21.15.7:0/1857883161 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5631a4cf5540 0x5631a4d09370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.802+0000 7f41f3035b80 1 --2- 172.21.15.7:0/1857883161 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631a4cf5ea0 0x5631a4d063d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.802+0000 7f41f3035b80 1 -- 172.21.15.7:0/1857883161 >> 172.21.15.7:0/1857883161 conn(0x5631a4c52c80 msgr2=0x5631a4cfdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:36.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.802+0000 7f41f3035b80 1 -- 172.21.15.7:0/1857883161 shutdown_connections 2023-07-20T02:51:36.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.802+0000 7f41f3035b80 1 -- 172.21.15.7:0/1857883161 wait complete. 2023-07-20T02:51:36.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.802+0000 7f41f3035b80 1 -- start start 2023-07-20T02:51:36.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.802+0000 7f41f3035b80 1 Processor -- start 2023-07-20T02:51:36.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.803+0000 7f41f3035b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631a4cf5540 0x5631a4d05990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:36.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.803+0000 7f41f3035b80 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631a4cf5ea0 0x5631a4d05ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:36.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.803+0000 7f41f3035b80 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5631a4cf6900 0x5631a4d61dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:36.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.803+0000 7f41f3035b80 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631a4d62310 con 0x5631a4cf5ea0 2023-07-20T02:51:36.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.803+0000 7f41f3035b80 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631a4d62480 con 0x5631a4cf6900 2023-07-20T02:51:36.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.803+0000 7f41f3035b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631a4d62640 con 0x5631a4cf5540 2023-07-20T02:51:36.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.803+0000 7f41f3035b80 10 client.0 _ll_register_callbacks cb 0x5631a4cf0890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-20T02:51:36.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.803+0000 7f41dd59b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631a4cf5540 0x5631a4d05990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:36.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.803+0000 7f41dd59b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631a4cf5540 0x5631a4d05990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.7:52758/0 (socket says 172.21.15.7:52758) 2023-07-20T02:51:36.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.803+0000 7f41dd59b700 1 -- 172.21.15.7:0/118163017 learned_addr learned my addr 172.21.15.7:0/118163017 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:36.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.803+0000 7f41dcd9a700 1 --2- 172.21.15.7:0/118163017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631a4cf5ea0 0x5631a4d05ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:36.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.803+0000 7f41ddd9c700 1 --2- 172.21.15.7:0/118163017 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5631a4cf6900 0x5631a4d61dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:36.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.803+0000 7f41dd59b700 1 -- 172.21.15.7:0/118163017 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5631a4cf6900 msgr2=0x5631a4d61dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-20T02:51:36.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.803+0000 7f41dd59b700 1 --2- 172.21.15.7:0/118163017 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5631a4cf6900 0x5631a4d61dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.803+0000 7f41dd59b700 1 -- 172.21.15.7:0/118163017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631a4cf5ea0 msgr2=0x5631a4d05ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-20T02:51:36.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.803+0000 7f41dd59b700 1 --2- 172.21.15.7:0/118163017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631a4cf5ea0 0x5631a4d05ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.803+0000 7f41dd59b700 1 -- 172.21.15.7:0/118163017 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5631a4a73d90 con 0x5631a4cf5540 2023-07-20T02:51:36.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.803+0000 7f41dcd9a700 1 --2- 172.21.15.7:0/118163017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631a4cf5ea0 0x5631a4d05ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T02:51:36.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.804+0000 7f41dd59b700 1 --2- 172.21.15.7:0/118163017 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631a4cf5540 0x5631a4d05990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f41d4096330 tx=0x7f41d4003710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b38d59a03bbbcf43 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:36.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.804+0000 7f41c0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2023-07-20T02:51:36.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.804+0000 7f41c0ff9700 1 -- 172.21.15.7:0/118163017 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d4096540 con 0x5631a4cf5540 2023-07-20T02:51:36.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.804+0000 7f41c0ff9700 1 -- 172.21.15.7:0/118163017 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f41d40b6070 con 0x5631a4cf5540 2023-07-20T02:51:36.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.804+0000 7f41f3035b80 1 -- 172.21.15.7:0/118163017 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f41c801e040 con 0x5631a4cf5540 2023-07-20T02:51:36.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.804+0000 7f41c0ff9700 1 -- 172.21.15.7:0/118163017 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f41d40966a0 con 0x5631a4cf5540 2023-07-20T02:51:36.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.804+0000 7f41f3035b80 10 client.24709 fetch_fsmap learned FSMap version 17 2023-07-20T02:51:36.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.804+0000 7f41f3035b80 1 -- 172.21.15.7:0/118163017 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x7f41d0009030 con 0x5631a4cf5540 2023-07-20T02:51:36.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.804+0000 7f41c0ff9700 1 -- 172.21.15.7:0/118163017 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f41d407c020 con 0x5631a4cf5540 2023-07-20T02:51:36.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.804+0000 7f41c0ff9700 1 -- 172.21.15.7:0/118163017 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 17) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f41d40a1df0 con 0x5631a4cf5540 2023-07-20T02:51:36.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.805+0000 7f41f3035b80 10 client.24709 fetch_fsmap finished waiting for FSMap version 17 2023-07-20T02:51:36.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.805+0000 7f41f3035b80 10 client.24709 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-20T02:51:36.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.805+0000 7f41f3035b80 20 client.24709 populate_metadata read hostname 'smithi007' 2023-07-20T02:51:36.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.805+0000 7f41f3035b80 1 --2- 172.21.15.7:0/118163017 >> [v2:172.21.15.158:6838/2552719389,v1:172.21.15.158:6839/2552719389] conn(0x5631a4d673e0 0x5631a4d63d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:36.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.805+0000 7f41f3035b80 4 client.24709 mds_command: new command op to 24572 tid=0[{"prefix": "status"}] 2023-07-20T02:51:36.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.805+0000 7f41f3035b80 1 -- 172.21.15.7:0/118163017 --> [v2:172.21.15.158:6838/2552719389,v1:172.21.15.158:6839/2552719389] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5631a4c3f730 con 0x5631a4d673e0 2023-07-20T02:51:36.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.805+0000 7f41dcd9a700 1 --2- 172.21.15.7:0/118163017 >> [v2:172.21.15.158:6838/2552719389,v1:172.21.15.158:6839/2552719389] conn(0x5631a4d673e0 0x5631a4d63d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:36.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.805+0000 7f41dcd9a700 1 --2- 172.21.15.7:0/118163017 >> [v2:172.21.15.158:6838/2552719389,v1:172.21.15.158:6839/2552719389] conn(0x5631a4d673e0 0x5631a4d63d30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f45a8e2ef8fbed1 server_cookie=db5d92394ae322a0 in_seq=0 out_seq=0 2023-07-20T02:51:36.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.805+0000 7f41c0ff9700 10 client.24709 ms_handle_connect on v2:172.21.15.158:6838/2552719389 2023-07-20T02:51:36.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.806+0000 7f41c0ff9700 1 -- 172.21.15.7:0/118163017 <== mds.0 v2:172.21.15.158:6838/2552719389 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x5631a4c3f730 con 0x5631a4d673e0 2023-07-20T02:51:36.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.806+0000 7f41c0ff9700 10 client.24709 handle_command_reply: tid=0 2023-07-20T02:51:36.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.806+0000 7f41f3035b80 1 client.24709 shutdown 2023-07-20T02:51:36.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.806+0000 7f41f3035b80 1 -- 172.21.15.7:0/118163017 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631a4cf5540 msgr2=0x5631a4d05990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:36.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.806+0000 7f41f3035b80 1 --2- 172.21.15.7:0/118163017 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631a4cf5540 0x5631a4d05990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f41d4096330 tx=0x7f41d4003710 comp rx=0 tx=0).stop 2023-07-20T02:51:36.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.806+0000 7f41f3035b80 1 -- 172.21.15.7:0/118163017 shutdown_connections 2023-07-20T02:51:36.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.806+0000 7f41f3035b80 1 --2- 172.21.15.7:0/118163017 >> [v2:172.21.15.158:6838/2552719389,v1:172.21.15.158:6839/2552719389] conn(0x5631a4d673e0 0x5631a4d63d30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.806+0000 7f41f3035b80 1 --2- 172.21.15.7:0/118163017 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x5631a4cf6900 0x5631a4d61dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.806+0000 7f41f3035b80 1 --2- 172.21.15.7:0/118163017 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5631a4cf5540 0x5631a4d05990 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.806+0000 7f41f3035b80 1 --2- 172.21.15.7:0/118163017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631a4cf5ea0 0x5631a4d05ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.806+0000 7f41c0ff9700 0 client.24709 ms_handle_reset on v2:172.21.15.158:6838/2552719389 2023-07-20T02:51:36.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.806+0000 7f41f3035b80 1 -- 172.21.15.7:0/118163017 >> 172.21.15.7:0/118163017 conn(0x5631a4c52c80 msgr2=0x5631a4d09370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:36.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.806+0000 7f41f3035b80 1 -- 172.21.15.7:0/118163017 shutdown_connections 2023-07-20T02:51:36.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.806+0000 7f41f3035b80 1 -- 172.21.15.7:0/118163017 wait complete. 2023-07-20T02:51:36.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.806+0000 7f41f3035b80 20 client.24709 trim_cache size 0 max 16384 2023-07-20T02:51:36.831 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-07-20T02:51:36.831 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_fsid": "1b04643c-26a7-11ee-9b34-001a4aab830c", 2023-07-20T02:51:36.831 INFO:teuthology.orchestra.run.smithi007.stdout: "whoami": 0, 2023-07-20T02:51:36.832 INFO:teuthology.orchestra.run.smithi007.stdout: "id": 24572, 2023-07-20T02:51:36.832 INFO:teuthology.orchestra.run.smithi007.stdout: "want_state": "up:active", 2023-07-20T02:51:36.832 INFO:teuthology.orchestra.run.smithi007.stdout: "state": "up:active", 2023-07-20T02:51:36.832 INFO:teuthology.orchestra.run.smithi007.stdout: "fs_name": "cephfs", 2023-07-20T02:51:36.832 INFO:teuthology.orchestra.run.smithi007.stdout: "rank_uptime": 7.662092328, 2023-07-20T02:51:36.832 INFO:teuthology.orchestra.run.smithi007.stdout: "mdsmap_epoch": 17, 2023-07-20T02:51:36.832 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch": 87, 2023-07-20T02:51:36.833 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch_barrier": 86, 2023-07-20T02:51:36.833 INFO:teuthology.orchestra.run.smithi007.stdout: "uptime": 13.860441872999999 2023-07-20T02:51:36.833 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-07-20T02:51:36.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.807+0000 7f41df7ff700 1 -- 172.21.15.7:0/4285386125 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f41c40a11d0 msgr2=0x7f41c40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:36.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.807+0000 7f41df7ff700 1 --2- 172.21.15.7:0/4285386125 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f41c40a11d0 0x7f41c40a3690 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f41d000f870 tx=0x7f41d000fc30 comp rx=0 tx=0).stop 2023-07-20T02:51:36.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.807+0000 7f41df7ff700 1 -- 172.21.15.7:0/4285386125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41d8100760 msgr2=0x7f41d8196470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:36.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.807+0000 7f41df7ff700 1 --2- 172.21.15.7:0/4285386125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41d8100760 0x7f41d8196470 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f41d400e970 tx=0x7f41d400ec80 comp rx=0 tx=0).stop 2023-07-20T02:51:36.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.807+0000 7f41df7ff700 1 -- 172.21.15.7:0/4285386125 shutdown_connections 2023-07-20T02:51:36.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.807+0000 7f41df7ff700 1 --2- 172.21.15.7:0/4285386125 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f41d81022f0 0x7f41d819da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.807+0000 7f41df7ff700 1 --2- 172.21.15.7:0/4285386125 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f41c40a11d0 0x7f41c40a3690 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.807+0000 7f41df7ff700 1 --2- 172.21.15.7:0/4285386125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41d8100760 0x7f41d8196470 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.807+0000 7f41df7ff700 1 --2- 172.21.15.7:0/4285386125 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41d8101950 0x7f41d81969b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:36.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.807+0000 7f41df7ff700 1 -- 172.21.15.7:0/4285386125 >> 172.21.15.7:0/4285386125 conn(0x7f41d80fbcb0 msgr2=0x7f41d80fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:36.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.807+0000 7f41df7ff700 1 -- 172.21.15.7:0/4285386125 shutdown_connections 2023-07-20T02:51:36.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:36.807+0000 7f41df7ff700 1 -- 172.21.15.7:0/4285386125 wait complete. 2023-07-20T02:51:36.838 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-07-20T02:51:36.839 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-07-20T02:51:36.840 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c -- ceph fs ls 2023-07-20T02:51:37.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:37 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/3045656772' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:51:37.564 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T02:51:37.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:37 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/3045656772' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:51:37.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:37 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/3045656772' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:51:38.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:38 smithi007 ceph-mon[126353]: pgmap v279: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 2 op/s 2023-07-20T02:51:38.463 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:38 smithi188 ceph-mon[129553]: pgmap v279: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 2 op/s 2023-07-20T02:51:38.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:38 smithi158 ceph-mon[128925]: pgmap v279: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 2 op/s 2023-07-20T02:51:39.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.084+0000 7fcb0418e700 1 -- 172.21.15.7:0/3706363984 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcafc106790 msgr2=0x7fcafc106b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:39.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.084+0000 7fcb0418e700 1 --2- 172.21.15.7:0/3706363984 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcafc106790 0x7fcafc106b70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcaf8009990 tx=0x7fcaf8009ca0 comp rx=0 tx=0).stop 2023-07-20T02:51:39.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.084+0000 7fcb0418e700 1 -- 172.21.15.7:0/3706363984 shutdown_connections 2023-07-20T02:51:39.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.084+0000 7fcb0418e700 1 --2- 172.21.15.7:0/3706363984 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcafc101150 0x7fcafc10d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:39.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.084+0000 7fcb0418e700 1 --2- 172.21.15.7:0/3706363984 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcafc106790 0x7fcafc106b70 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:39.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.084+0000 7fcb0418e700 1 --2- 172.21.15.7:0/3706363984 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcafc100790 0x7fcafc100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:39.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.084+0000 7fcb0418e700 1 -- 172.21.15.7:0/3706363984 >> 172.21.15.7:0/3706363984 conn(0x7fcafc0fc260 msgr2=0x7fcafc0fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:39.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.084+0000 7fcb0418e700 1 -- 172.21.15.7:0/3706363984 shutdown_connections 2023-07-20T02:51:39.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.084+0000 7fcb0418e700 1 -- 172.21.15.7:0/3706363984 wait complete. 2023-07-20T02:51:39.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.085+0000 7fcb0418e700 1 Processor -- start 2023-07-20T02:51:39.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.085+0000 7fcb0418e700 1 -- start start 2023-07-20T02:51:39.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.085+0000 7fcb0418e700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcafc100790 0x7fcafc19c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:39.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.085+0000 7fcb0418e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcafc101150 0x7fcafc19cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:39.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.086+0000 7fcb0418e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcafc106790 0x7fcafc196710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:39.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.086+0000 7fcb0418e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc196d80 con 0x7fcafc101150 2023-07-20T02:51:39.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.086+0000 7fcb0418e700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc196ef0 con 0x7fcafc100790 2023-07-20T02:51:39.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.086+0000 7fcb0418e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc1970b0 con 0x7fcafc106790 2023-07-20T02:51:39.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.086+0000 7fcb01729700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcafc101150 0x7fcafc19cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:39.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.086+0000 7fcb01729700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcafc101150 0x7fcafc19cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52010/0 (socket says 172.21.15.7:52010) 2023-07-20T02:51:39.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.086+0000 7fcb01729700 1 -- 172.21.15.7:0/2127573110 learned_addr learned my addr 172.21.15.7:0/2127573110 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:39.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.086+0000 7fcb01f2a700 1 --2- 172.21.15.7:0/2127573110 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcafc100790 0x7fcafc19c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:39.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.086+0000 7fcb0272b700 1 --2- 172.21.15.7:0/2127573110 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcafc106790 0x7fcafc196710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:39.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.086+0000 7fcb01729700 1 -- 172.21.15.7:0/2127573110 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcafc106790 msgr2=0x7fcafc196710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:39.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.086+0000 7fcb01729700 1 --2- 172.21.15.7:0/2127573110 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcafc106790 0x7fcafc196710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:39.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.086+0000 7fcb01729700 1 -- 172.21.15.7:0/2127573110 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcafc100790 msgr2=0x7fcafc19c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:39.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.086+0000 7fcb01729700 1 --2- 172.21.15.7:0/2127573110 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcafc100790 0x7fcafc19c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:39.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.086+0000 7fcb01729700 1 -- 172.21.15.7:0/2127573110 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf80095f0 con 0x7fcafc101150 2023-07-20T02:51:39.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.087+0000 7fcb01729700 1 --2- 172.21.15.7:0/2127573110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcafc101150 0x7fcafc19cbd0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fcaec00d980 tx=0x7fcaec00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:39.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.087+0000 7fcaf2ffd700 1 -- 172.21.15.7:0/2127573110 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaec004220 con 0x7fcafc101150 2023-07-20T02:51:39.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.087+0000 7fcaf2ffd700 1 -- 172.21.15.7:0/2127573110 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcaec007640 con 0x7fcafc101150 2023-07-20T02:51:39.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.087+0000 7fcb0418e700 1 -- 172.21.15.7:0/2127573110 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcafc197390 con 0x7fcafc101150 2023-07-20T02:51:39.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.087+0000 7fcaf2ffd700 1 -- 172.21.15.7:0/2127573110 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaec005690 con 0x7fcafc101150 2023-07-20T02:51:39.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.087+0000 7fcb0418e700 1 -- 172.21.15.7:0/2127573110 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcafc1a3680 con 0x7fcafc101150 2023-07-20T02:51:39.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.088+0000 7fcb0418e700 1 -- 172.21.15.7:0/2127573110 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcafc04e470 con 0x7fcafc101150 2023-07-20T02:51:39.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.090+0000 7fcaf2ffd700 1 -- 172.21.15.7:0/2127573110 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fcaec004750 con 0x7fcafc101150 2023-07-20T02:51:39.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.091+0000 7fcaf2ffd700 1 --2- 172.21.15.7:0/2127573110 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fcae80a11d0 0x7fcae80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:39.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.091+0000 7fcaf2ffd700 1 -- 172.21.15.7:0/2127573110 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fcaec0cc8b0 con 0x7fcafc101150 2023-07-20T02:51:39.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.091+0000 7fcb01f2a700 1 --2- 172.21.15.7:0/2127573110 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fcae80a11d0 0x7fcae80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:39.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.092+0000 7fcb01f2a700 1 --2- 172.21.15.7:0/2127573110 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fcae80a11d0 0x7fcae80a3690 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcaf800b2e0 tx=0x7fcaf8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:39.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.095+0000 7fcaf2ffd700 1 -- 172.21.15.7:0/2127573110 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcaec095db0 con 0x7fcafc101150 2023-07-20T02:51:39.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.346+0000 7fcb0418e700 1 -- 172.21.15.7:0/2127573110 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fcafc1a3970 con 0x7fcafc101150 2023-07-20T02:51:39.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.346+0000 7fcaf2ffd700 1 -- 172.21.15.7:0/2127573110 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+88 (secure 0 0 0) 0x7fcaec016020 con 0x7fcafc101150 2023-07-20T02:51:39.361 INFO:teuthology.orchestra.run.smithi007.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-07-20T02:51:39.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.348+0000 7fcb0418e700 1 -- 172.21.15.7:0/2127573110 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fcae80a11d0 msgr2=0x7fcae80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:39.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.348+0000 7fcb0418e700 1 --2- 172.21.15.7:0/2127573110 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fcae80a11d0 0x7fcae80a3690 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcaf800b2e0 tx=0x7fcaf8005fb0 comp rx=0 tx=0).stop 2023-07-20T02:51:39.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.349+0000 7fcb0418e700 1 -- 172.21.15.7:0/2127573110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcafc101150 msgr2=0x7fcafc19cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:39.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.349+0000 7fcb0418e700 1 --2- 172.21.15.7:0/2127573110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcafc101150 0x7fcafc19cbd0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fcaec00d980 tx=0x7fcaec00dc90 comp rx=0 tx=0).stop 2023-07-20T02:51:39.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.349+0000 7fcb0418e700 1 -- 172.21.15.7:0/2127573110 shutdown_connections 2023-07-20T02:51:39.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.349+0000 7fcb0418e700 1 --2- 172.21.15.7:0/2127573110 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcafc106790 0x7fcafc196710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:39.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.349+0000 7fcb0418e700 1 --2- 172.21.15.7:0/2127573110 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fcafc100790 0x7fcafc19c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:39.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.349+0000 7fcb0418e700 1 --2- 172.21.15.7:0/2127573110 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fcae80a11d0 0x7fcae80a3690 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:39.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.349+0000 7fcb0418e700 1 --2- 172.21.15.7:0/2127573110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcafc101150 0x7fcafc19cbd0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:39.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.349+0000 7fcb0418e700 1 -- 172.21.15.7:0/2127573110 >> 172.21.15.7:0/2127573110 conn(0x7fcafc0fc260 msgr2=0x7fcafc0fc910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:39.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.349+0000 7fcb0418e700 1 -- 172.21.15.7:0/2127573110 shutdown_connections 2023-07-20T02:51:39.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:39.349+0000 7fcb0418e700 1 -- 172.21.15.7:0/2127573110 wait complete. 2023-07-20T02:51:39.638 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:39 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/2127573110' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-20T02:51:39.714 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:39 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/2127573110' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-20T02:51:39.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:39 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/2127573110' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-20T02:51:39.942 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-07-20T02:51:39.942 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-07-20T02:51:39.942 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi007.front.sepia.ceph.com 2023-07-20T02:51:39.943 INFO:tasks.cephfs.mount:self.client.name = client.admin 2023-07-20T02:51:39.943 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-07-20T02:51:39.943 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-07-20T02:51:39.943 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-07-20T02:51:39.943 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-07-20T02:51:39.944 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2023-07-20T02:51:39.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T02:51:39.944 DEBUG:teuthology.orchestra.run.smithi007:> ip addr 2023-07-20T02:51:39.990 INFO:teuthology.orchestra.run.smithi007.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-07-20T02:51:39.990 INFO:teuthology.orchestra.run.smithi007.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-07-20T02:51:39.990 INFO:teuthology.orchestra.run.smithi007.stdout: inet 127.0.0.1/8 scope host lo 2023-07-20T02:51:39.990 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2023-07-20T02:51:39.990 INFO:teuthology.orchestra.run.smithi007.stdout: inet6 ::1/128 scope host 2023-07-20T02:51:39.990 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2023-07-20T02:51:39.991 INFO:teuthology.orchestra.run.smithi007.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-20T02:51:39.991 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:6b:fd:dc brd ff:ff:ff:ff:ff:ff 2023-07-20T02:51:39.991 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp7s0f0 2023-07-20T02:51:39.991 INFO:teuthology.orchestra.run.smithi007.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-20T02:51:39.991 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:6b:fd:dd brd ff:ff:ff:ff:ff:ff 2023-07-20T02:51:39.991 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp7s0f1 2023-07-20T02:51:39.991 INFO:teuthology.orchestra.run.smithi007.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-07-20T02:51:39.992 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:bd:15:ea brd ff:ff:ff:ff:ff:ff 2023-07-20T02:51:39.992 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp1s0f0 2023-07-20T02:51:39.992 INFO:teuthology.orchestra.run.smithi007.stdout: inet 172.21.15.7/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-07-20T02:51:39.992 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft 41471sec preferred_lft 41471sec 2023-07-20T02:51:39.992 INFO:teuthology.orchestra.run.smithi007.stdout: inet6 fe80::ec4:7aff:febd:15ea/64 scope link 2023-07-20T02:51:39.992 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2023-07-20T02:51:39.993 INFO:teuthology.orchestra.run.smithi007.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-20T02:51:39.993 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:bd:15:eb brd ff:ff:ff:ff:ff:ff 2023-07-20T02:51:39.993 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp1s0f1 2023-07-20T02:51:39.994 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-07-20T02:51:39.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T02:51:39.994 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec bash -c ' 2023-07-20T02:51:39.995 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2023-07-20T02:51:39.995 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link add name ceph-brx type bridge 2023-07-20T02:51:39.995 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip addr flush dev ceph-brx 2023-07-20T02:51:39.995 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link set ceph-brx up 2023-07-20T02:51:39.996 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-07-20T02:51:39.996 DEBUG:teuthology.orchestra.run.smithi007:> ') 2023-07-20T02:51:40.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T02:51:40.476 DEBUG:teuthology.orchestra.run.smithi007:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-07-20T02:51:40.548 INFO:teuthology.orchestra.run.smithi007.stdout:1 2023-07-20T02:51:40.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T02:51:40.553 DEBUG:teuthology.orchestra.run.smithi007:> route 2023-07-20T02:51:40.674 INFO:teuthology.orchestra.run.smithi007.stdout:Kernel IP routing table 2023-07-20T02:51:40.674 INFO:teuthology.orchestra.run.smithi007.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-07-20T02:51:40.674 INFO:teuthology.orchestra.run.smithi007.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-07-20T02:51:40.675 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-07-20T02:51:40.675 INFO:teuthology.orchestra.run.smithi007.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-07-20T02:51:40.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T02:51:40.676 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec bash -c ' 2023-07-20T02:51:40.677 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2023-07-20T02:51:40.677 DEBUG:teuthology.orchestra.run.smithi007:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-07-20T02:51:40.677 DEBUG:teuthology.orchestra.run.smithi007:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-07-20T02:51:40.677 DEBUG:teuthology.orchestra.run.smithi007:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-07-20T02:51:40.678 DEBUG:teuthology.orchestra.run.smithi007:> ') 2023-07-20T02:51:40.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:40 smithi007 ceph-mon[126353]: pgmap v280: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 416 B/s wr, 0 op/s 2023-07-20T02:51:40.713 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:40 smithi188 ceph-mon[129553]: pgmap v280: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 416 B/s wr, 0 op/s 2023-07-20T02:51:40.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:40 smithi158 ceph-mon[128925]: pgmap v280: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 416 B/s wr, 0 op/s 2023-07-20T02:51:41.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T02:51:41.384 DEBUG:teuthology.orchestra.run.smithi007:> ip netns list 2023-07-20T02:51:41.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T02:51:41.469 DEBUG:teuthology.orchestra.run.smithi007:> ip netns list-id 2023-07-20T02:51:41.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T02:51:41.553 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec bash -c ' 2023-07-20T02:51:41.553 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2023-07-20T02:51:41.554 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-07-20T02:51:41.554 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2023-07-20T02:51:41.554 DEBUG:teuthology.orchestra.run.smithi007:> ') 2023-07-20T02:51:41.743 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2023-07-20T02:51:41.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T02:51:41.744 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec bash -c ' 2023-07-20T02:51:41.744 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2023-07-20T02:51:41.744 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2023-07-20T02:51:41.744 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-07-20T02:51:41.745 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2023-07-20T02:51:41.745 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2023-07-20T02:51:41.745 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2023-07-20T02:51:41.745 DEBUG:teuthology.orchestra.run.smithi007:> ') 2023-07-20T02:51:42.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T02:51:42.026 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec bash -c ' 2023-07-20T02:51:42.026 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2023-07-20T02:51:42.027 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link set brx.0 up 2023-07-20T02:51:42.027 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link set dev brx.0 master ceph-brx 2023-07-20T02:51:42.027 DEBUG:teuthology.orchestra.run.smithi007:> ') 2023-07-20T02:51:42.206 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2023-07-20T02:51:42.206 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-20T02:51:42.206 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2023-07-20T02:51:42.250 INFO:teuthology.orchestra.run.smithi007.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2023-07-20T02:51:42.251 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-20T02:51:42.251 DEBUG:teuthology.orchestra.run.smithi007:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2023-07-20T02:51:42.335 DEBUG:teuthology.orchestra.run.smithi007:> sudo modprobe fuse 2023-07-20T02:51:42.448 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-07-20T02:51:42.449 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-07-20T02:51:42.519 INFO:teuthology.orchestra.run.smithi007.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-07-20T02:51:42.522 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-07-20T02:51:42.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T02:51:42.523 DEBUG:teuthology.orchestra.run.smithi007:> ls /sys/fs/fuse/connections 2023-07-20T02:51:42.566 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-07-20T02:51:42.567 DEBUG:teuthology.orchestra.run.smithi007:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2023-07-20T02:51:42.611 DEBUG:teuthology.orchestra.run.smithi007:> sudo modprobe fuse 2023-07-20T02:51:42.681 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-07-20T02:51:42.682 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-07-20T02:51:42.752 INFO:teuthology.orchestra.run.smithi007.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-07-20T02:51:42.755 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-07-20T02:51:42.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T02:51:42.756 DEBUG:teuthology.orchestra.run.smithi007:> ls /sys/fs/fuse/connections 2023-07-20T02:51:42.815 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi007.stderr:2023-07-20T02:51:42.801+0000 7f8d8c9175c0 -1 init, newargv = 0x55b9c75138a0 newargc=15 2023-07-20T02:51:42.816 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi007.stderr:ceph-fuse[163864]: starting ceph client 2023-07-20T02:51:42.825 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi007.stderr:ceph-fuse[163864]: starting fuse 2023-07-20T02:51:42.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:42 smithi158 ceph-mon[128925]: pgmap v281: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-20T02:51:42.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:42 smithi007 ceph-mon[126353]: pgmap v281: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-20T02:51:42.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:42 smithi188 ceph-mon[129553]: pgmap v281: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-20T02:51:43.901 DEBUG:teuthology.orchestra.run.smithi007:> sudo modprobe fuse 2023-07-20T02:51:43.980 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-07-20T02:51:43.980 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-07-20T02:51:44.051 INFO:teuthology.orchestra.run.smithi007.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-07-20T02:51:44.054 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-07-20T02:51:44.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T02:51:44.056 DEBUG:teuthology.orchestra.run.smithi007:> ls /sys/fs/fuse/connections 2023-07-20T02:51:44.100 INFO:teuthology.orchestra.run.smithi007.stdout:121 2023-07-20T02:51:44.100 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [121] 2023-07-20T02:51:44.101 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-07-20T02:51:44.101 DEBUG:teuthology.orchestra.run.smithi007:> import glob 2023-07-20T02:51:44.101 DEBUG:teuthology.orchestra.run.smithi007:> import re 2023-07-20T02:51:44.102 DEBUG:teuthology.orchestra.run.smithi007:> import os 2023-07-20T02:51:44.102 DEBUG:teuthology.orchestra.run.smithi007:> import subprocess 2023-07-20T02:51:44.102 DEBUG:teuthology.orchestra.run.smithi007:> 2023-07-20T02:51:44.103 DEBUG:teuthology.orchestra.run.smithi007:> def _find_admin_socket(client_name): 2023-07-20T02:51:44.103 DEBUG:teuthology.orchestra.run.smithi007:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2023-07-20T02:51:44.103 DEBUG:teuthology.orchestra.run.smithi007:> files = glob.glob(asok_path) 2023-07-20T02:51:44.104 DEBUG:teuthology.orchestra.run.smithi007:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2023-07-20T02:51:44.104 DEBUG:teuthology.orchestra.run.smithi007:> 2023-07-20T02:51:44.104 DEBUG:teuthology.orchestra.run.smithi007:> # Given a non-glob path, it better be there 2023-07-20T02:51:44.105 DEBUG:teuthology.orchestra.run.smithi007:> if "*" not in asok_path: 2023-07-20T02:51:44.105 DEBUG:teuthology.orchestra.run.smithi007:> assert(len(files) == 1) 2023-07-20T02:51:44.105 DEBUG:teuthology.orchestra.run.smithi007:> return files[0] 2023-07-20T02:51:44.105 DEBUG:teuthology.orchestra.run.smithi007:> 2023-07-20T02:51:44.106 DEBUG:teuthology.orchestra.run.smithi007:> for f in files: 2023-07-20T02:51:44.106 DEBUG:teuthology.orchestra.run.smithi007:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-07-20T02:51:44.107 DEBUG:teuthology.orchestra.run.smithi007:> if os.path.exists("/proc/{0}".format(pid)): 2023-07-20T02:51:44.107 DEBUG:teuthology.orchestra.run.smithi007:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-07-20T02:51:44.108 DEBUG:teuthology.orchestra.run.smithi007:> contents = proc_f.read() 2023-07-20T02:51:44.109 DEBUG:teuthology.orchestra.run.smithi007:> if mountpoint in contents: 2023-07-20T02:51:44.109 DEBUG:teuthology.orchestra.run.smithi007:> return f 2023-07-20T02:51:44.110 DEBUG:teuthology.orchestra.run.smithi007:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-07-20T02:51:44.111 DEBUG:teuthology.orchestra.run.smithi007:> 2023-07-20T02:51:44.112 DEBUG:teuthology.orchestra.run.smithi007:> print(_find_admin_socket("client.admin")) 2023-07-20T02:51:44.112 DEBUG:teuthology.orchestra.run.smithi007:> ' 2023-07-20T02:51:44.287 INFO:teuthology.orchestra.run.smithi007.stdout:/var/run/ceph/ceph-client.admin.163864.asok 2023-07-20T02:51:44.464 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.163864.asok 2023-07-20T02:51:44.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T02:51:44.466 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.163864.asok status 2023-07-20T02:51:44.697 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-07-20T02:51:44.697 INFO:teuthology.orchestra.run.smithi007.stdout: "metadata": { 2023-07-20T02:51:44.697 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_sha1": "1b0b2eb005a7c51d97a788b9900d511e1394b537", 2023-07-20T02:51:44.697 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_version": "ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev)", 2023-07-20T02:51:44.697 INFO:teuthology.orchestra.run.smithi007.stdout: "entity_id": "admin", 2023-07-20T02:51:44.698 INFO:teuthology.orchestra.run.smithi007.stdout: "hostname": "smithi007", 2023-07-20T02:51:44.698 INFO:teuthology.orchestra.run.smithi007.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2023-07-20T02:51:44.698 INFO:teuthology.orchestra.run.smithi007.stdout: "pid": "163864", 2023-07-20T02:51:44.698 INFO:teuthology.orchestra.run.smithi007.stdout: "root": "/" 2023-07-20T02:51:44.698 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-07-20T02:51:44.698 INFO:teuthology.orchestra.run.smithi007.stdout: "dentry_count": 0, 2023-07-20T02:51:44.698 INFO:teuthology.orchestra.run.smithi007.stdout: "dentry_pinned_count": 0, 2023-07-20T02:51:44.698 INFO:teuthology.orchestra.run.smithi007.stdout: "id": 24721, 2023-07-20T02:51:44.699 INFO:teuthology.orchestra.run.smithi007.stdout: "inst": { 2023-07-20T02:51:44.699 INFO:teuthology.orchestra.run.smithi007.stdout: "name": { 2023-07-20T02:51:44.699 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "client", 2023-07-20T02:51:44.699 INFO:teuthology.orchestra.run.smithi007.stdout: "num": 24721 2023-07-20T02:51:44.699 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-07-20T02:51:44.699 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": { 2023-07-20T02:51:44.699 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v1", 2023-07-20T02:51:44.699 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:0", 2023-07-20T02:51:44.700 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2635335317 2023-07-20T02:51:44.700 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-07-20T02:51:44.700 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-07-20T02:51:44.700 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": { 2023-07-20T02:51:44.700 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v1", 2023-07-20T02:51:44.700 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:0", 2023-07-20T02:51:44.700 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2635335317 2023-07-20T02:51:44.701 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-07-20T02:51:44.701 INFO:teuthology.orchestra.run.smithi007.stdout: "inst_str": "client.24721 172.21.15.7:0/2635335317", 2023-07-20T02:51:44.701 INFO:teuthology.orchestra.run.smithi007.stdout: "addr_str": "172.21.15.7:0/2635335317", 2023-07-20T02:51:44.701 INFO:teuthology.orchestra.run.smithi007.stdout: "inode_count": 1, 2023-07-20T02:51:44.701 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_epoch": 17, 2023-07-20T02:51:44.701 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_epoch": 87, 2023-07-20T02:51:44.701 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_epoch_barrier": 0, 2023-07-20T02:51:44.701 INFO:teuthology.orchestra.run.smithi007.stdout: "blocklisted": false, 2023-07-20T02:51:44.702 INFO:teuthology.orchestra.run.smithi007.stdout: "fs_name": "cephfs" 2023-07-20T02:51:44.702 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-07-20T02:51:44.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T02:51:44.711 DEBUG:teuthology.orchestra.run.smithi007:> stat --file-system '--printf=%T 2023-07-20T02:51:44.712 DEBUG:teuthology.orchestra.run.smithi007:> ' -- /home/ubuntu/cephtest/mnt.admin 2023-07-20T02:51:44.757 INFO:teuthology.orchestra.run.smithi007.stdout:fuseblk 2023-07-20T02:51:44.757 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2023-07-20T02:51:44.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T02:51:44.758 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2023-07-20T02:51:44.859 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:44 smithi007 ceph-mon[126353]: pgmap v282: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-20T02:51:44.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T02:51:44.871 DEBUG:teuthology.orchestra.run.smithi007:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2023-07-20T02:51:44.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:44 smithi158 ceph-mon[128925]: pgmap v282: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-20T02:51:44.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:44 smithi188 ceph-mon[129553]: pgmap v282: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-20T02:51:44.992 INFO:teuthology.orchestra.run.smithi007.stdout:# file: . 2023-07-20T02:51:44.992 INFO:teuthology.orchestra.run.smithi007.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2023-07-20T02:51:44.993 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:51:44.994 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-07-20T02:51:44.994 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-20T02:51:45.040 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T02:51:45.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.276+0000 7f3c6a1ff700 1 -- 172.21.15.7:0/1459602897 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c64102300 msgr2=0x7f3c64108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:45.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.276+0000 7f3c6a1ff700 1 --2- 172.21.15.7:0/1459602897 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c64102300 0x7f3c64108bb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c00b0d0 tx=0x7f3c5c00b3e0 comp rx=0 tx=0).stop 2023-07-20T02:51:45.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.276+0000 7f3c6a1ff700 1 -- 172.21.15.7:0/1459602897 shutdown_connections 2023-07-20T02:51:45.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.276+0000 7f3c6a1ff700 1 --2- 172.21.15.7:0/1459602897 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c64102300 0x7f3c64108bb0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:45.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.276+0000 7f3c6a1ff700 1 --2- 172.21.15.7:0/1459602897 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c64100770 0x7f3c64100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:45.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.276+0000 7f3c6a1ff700 1 --2- 172.21.15.7:0/1459602897 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c64101960 0x7f3c64101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:45.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.276+0000 7f3c6a1ff700 1 -- 172.21.15.7:0/1459602897 >> 172.21.15.7:0/1459602897 conn(0x7f3c640fbca0 msgr2=0x7f3c640fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:45.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.276+0000 7f3c6a1ff700 1 -- 172.21.15.7:0/1459602897 shutdown_connections 2023-07-20T02:51:45.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.276+0000 7f3c6a1ff700 1 -- 172.21.15.7:0/1459602897 wait complete. 2023-07-20T02:51:45.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.277+0000 7f3c6a1ff700 1 Processor -- start 2023-07-20T02:51:45.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.278+0000 7f3c6a1ff700 1 -- start start 2023-07-20T02:51:45.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.278+0000 7f3c6a1ff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c64100770 0x7f3c64074520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:45.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.278+0000 7f3c6a1ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c64101960 0x7f3c64072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:45.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.278+0000 7f3c6a1ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c64102300 0x7f3c640730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:45.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.278+0000 7f3c6a1ff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c64073840 con 0x7f3c64101960 2023-07-20T02:51:45.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.278+0000 7f3c6a1ff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c640739b0 con 0x7f3c64100770 2023-07-20T02:51:45.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.278+0000 7f3c6a1ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c641a0520 con 0x7f3c64102300 2023-07-20T02:51:45.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.278+0000 7f3c62ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c64101960 0x7f3c64072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:45.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.278+0000 7f3c62ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c64101960 0x7f3c64072b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48476/0 (socket says 172.21.15.7:48476) 2023-07-20T02:51:45.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.278+0000 7f3c62ffd700 1 -- 172.21.15.7:0/3716987033 learned_addr learned my addr 172.21.15.7:0/3716987033 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:45.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.278+0000 7f3c637fe700 1 --2- 172.21.15.7:0/3716987033 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c64100770 0x7f3c64074520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:45.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.278+0000 7f3c63fff700 1 --2- 172.21.15.7:0/3716987033 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c64102300 0x7f3c640730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:45.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.279+0000 7f3c62ffd700 1 -- 172.21.15.7:0/3716987033 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c64102300 msgr2=0x7f3c640730a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:45.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.279+0000 7f3c62ffd700 1 --2- 172.21.15.7:0/3716987033 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c64102300 0x7f3c640730a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:45.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.279+0000 7f3c62ffd700 1 -- 172.21.15.7:0/3716987033 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c64100770 msgr2=0x7f3c64074520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:45.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.279+0000 7f3c62ffd700 1 --2- 172.21.15.7:0/3716987033 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c64100770 0x7f3c64074520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:45.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.279+0000 7f3c62ffd700 1 -- 172.21.15.7:0/3716987033 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5c009d00 con 0x7f3c64101960 2023-07-20T02:51:45.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.279+0000 7f3c637fe700 1 --2- 172.21.15.7:0/3716987033 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c64100770 0x7f3c64074520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T02:51:45.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.279+0000 7f3c62ffd700 1 --2- 172.21.15.7:0/3716987033 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c64101960 0x7f3c64072b60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3c5800e9a0 tx=0x7f3c5800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:45.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.279+0000 7f3c60ff9700 1 -- 172.21.15.7:0/3716987033 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c58009790 con 0x7f3c64101960 2023-07-20T02:51:45.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.279+0000 7f3c6a1ff700 1 -- 172.21.15.7:0/3716987033 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c641a0800 con 0x7f3c64101960 2023-07-20T02:51:45.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.279+0000 7f3c60ff9700 1 -- 172.21.15.7:0/3716987033 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3c5800cbd0 con 0x7f3c64101960 2023-07-20T02:51:45.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.279+0000 7f3c60ff9700 1 -- 172.21.15.7:0/3716987033 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c58018680 con 0x7f3c64101960 2023-07-20T02:51:45.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.279+0000 7f3c6a1ff700 1 -- 172.21.15.7:0/3716987033 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c641a0d10 con 0x7f3c64101960 2023-07-20T02:51:45.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.280+0000 7f3c6a1ff700 1 -- 172.21.15.7:0/3716987033 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c64107c70 con 0x7f3c64101960 2023-07-20T02:51:45.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.282+0000 7f3c60ff9700 1 -- 172.21.15.7:0/3716987033 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f3c5800cd40 con 0x7f3c64101960 2023-07-20T02:51:45.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.284+0000 7f3c60ff9700 1 --2- 172.21.15.7:0/3716987033 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f3c4c0a11a0 0x7f3c4c0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:45.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.284+0000 7f3c60ff9700 1 -- 172.21.15.7:0/3716987033 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f3c58014070 con 0x7f3c64101960 2023-07-20T02:51:45.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.284+0000 7f3c637fe700 1 --2- 172.21.15.7:0/3716987033 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f3c4c0a11a0 0x7f3c4c0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:45.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.285+0000 7f3c637fe700 1 --2- 172.21.15.7:0/3716987033 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f3c4c0a11a0 0x7f3c4c0a3660 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3c641017c0 tx=0x7f3c54009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:45.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.288+0000 7f3c60ff9700 1 -- 172.21.15.7:0/3716987033 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3c58051780 con 0x7f3c64101960 2023-07-20T02:51:45.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.497+0000 7f3c6a1ff700 1 -- 172.21.15.7:0/3716987033 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c6402c970 con 0x7f3c64101960 2023-07-20T02:51:45.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.498+0000 7f3c60ff9700 1 -- 172.21.15.7:0/3716987033 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+23927 (secure 0 0 0) 0x7f3c580963e0 con 0x7f3c64101960 2023-07-20T02:51:45.513 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:51:45.513 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":87,"fsid":"1b04643c-26a7-11ee-9b34-001a4aab830c","created":"2023-07-20T02:43:56.544719+0000","modified":"2023-07-20T02:51:33.697865+0000","last_up_change":"2023-07-20T02:50:40.126926+0000","last_in_change":"2023-07-20T02:50:21.875020+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T02:47:12.055539+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-20T02:51:26.617961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-20T02:51:28.081016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-20T02:51:31.658088+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ad6c6339-6e23-4c21-9fde-2dbecdafdb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6803","nonce":3322992091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6805","nonce":3322992091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6809","nonce":3322992091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6807","nonce":3322992091}]},"public_addr":"172.21.15.7:6803/3322992091","cluster_addr":"172.21.15.7:6805/3322992091","heartbeat_back_addr":"172.21.15.7:6809/3322992091","heartbeat_front_addr":"172.21.15.7:6807/3322992091","state":["exists","up"]},{"osd":1,"uuid":"cad207b3-a1ea-4f07-bd94-9b11fd573a69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6801","nonce":2796771612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6803","nonce":2796771612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6807","nonce":2796771612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6805","nonce":2796771612}]},"public_addr":"172.21.15.158:6801/2796771612","cluster_addr":"172.21.15.158:6803/2796771612","heartbeat_back_addr":"172.21.15.158:6807/2796771612","heartbeat_front_addr":"172.21.15.158:6805/2796771612","state":["exists","up"]},{"osd":2,"uuid":"2c777fe6-d003-49d0-af37-f63fca37ff06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6801","nonce":2928307258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6803","nonce":2928307258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6807","nonce":2928307258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6805","nonce":2928307258}]},"public_addr":"172.21.15.188:6801/2928307258","cluster_addr":"172.21.15.188:6803/2928307258","heartbeat_back_addr":"172.21.15.188:6807/2928307258","heartbeat_front_addr":"172.21.15.188:6805/2928307258","state":["exists","up"]},{"osd":3,"uuid":"59e29965-018f-4eb6-adbd-1a9e1df84245","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6811","nonce":2192953827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6813","nonce":2192953827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6817","nonce":2192953827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6815","nonce":2192953827}]},"public_addr":"172.21.15.7:6811/2192953827","cluster_addr":"172.21.15.7:6813/2192953827","heartbeat_back_addr":"172.21.15.7:6817/2192953827","heartbeat_front_addr":"172.21.15.7:6815/2192953827","state":["exists","up"]},{"osd":4,"uuid":"e2453384-3b09-4f2a-bea7-172d8fc466d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6809","nonce":3330591132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6810","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6811","nonce":3330591132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6814","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6815","nonce":3330591132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6813","nonce":3330591132}]},"public_addr":"172.21.15.158:6809/3330591132","cluster_addr":"172.21.15.158:6811/3330591132","heartbeat_back_addr":"172.21.15.158:6815/3330591132","heartbeat_front_addr":"172.21.15.158:6813/3330591132","state":["exists","up"]},{"osd":5,"uuid":"c3c04c13-0ce5-440d-9e62-0e301ad9f771","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6809","nonce":3142012805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6811","nonce":3142012805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6815","nonce":3142012805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6813","nonce":3142012805}]},"public_addr":"172.21.15.188:6809/3142012805","cluster_addr":"172.21.15.188:6811/3142012805","heartbeat_back_addr":"172.21.15.188:6815/3142012805","heartbeat_front_addr":"172.21.15.188:6813/3142012805","state":["exists","up"]},{"osd":6,"uuid":"82ef2d33-1045-41f1-bf69-b03404bcace8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6819","nonce":859560576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6821","nonce":859560576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6825","nonce":859560576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6823","nonce":859560576}]},"public_addr":"172.21.15.7:6819/859560576","cluster_addr":"172.21.15.7:6821/859560576","heartbeat_back_addr":"172.21.15.7:6825/859560576","heartbeat_front_addr":"172.21.15.7:6823/859560576","state":["exists","up"]},{"osd":7,"uuid":"020ec859-3fc2-4da2-a0dc-10530a4a366c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6817","nonce":2578758635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6819","nonce":2578758635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6822","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6823","nonce":2578758635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6821","nonce":2578758635}]},"public_addr":"172.21.15.158:6817/2578758635","cluster_addr":"172.21.15.158:6819/2578758635","heartbeat_back_addr":"172.21.15.158:6823/2578758635","heartbeat_front_addr":"172.21.15.158:6821/2578758635","state":["exists","up"]},{"osd":8,"uuid":"4081496a-45ee-4b25-8f17-a23665521187","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6817","nonce":3020824843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6819","nonce":3020824843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6823","nonce":3020824843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6821","nonce":3020824843}]},"public_addr":"172.21.15.188:6817/3020824843","cluster_addr":"172.21.15.188:6819/3020824843","heartbeat_back_addr":"172.21.15.188:6823/3020824843","heartbeat_front_addr":"172.21.15.188:6821/3020824843","state":["exists","up"]},{"osd":9,"uuid":"723580ae-efe3-4d86-95fd-d7a4c0631425","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6827","nonce":1612464825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6829","nonce":1612464825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6833","nonce":1612464825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6831","nonce":1612464825}]},"public_addr":"172.21.15.7:6827/1612464825","cluster_addr":"172.21.15.7:6829/1612464825","heartbeat_back_addr":"172.21.15.7:6833/1612464825","heartbeat_front_addr":"172.21.15.7:6831/1612464825","state":["exists","up"]},{"osd":10,"uuid":"c2651548-fe46-4d2d-b911-278036412ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6825","nonce":1653166210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6827","nonce":1653166210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6831","nonce":1653166210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6829","nonce":1653166210}]},"public_addr":"172.21.15.158:6825/1653166210","cluster_addr":"172.21.15.158:6827/1653166210","heartbeat_back_addr":"172.21.15.158:6831/1653166210","heartbeat_front_addr":"172.21.15.158:6829/1653166210","state":["exists","up"]},{"osd":11,"uuid":"f7262bcb-4243-403a-a321-fc2bdf1d4d25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6825","nonce":1230196525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6827","nonce":1230196525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6831","nonce":1230196525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6829","nonce":1230196525}]},"public_addr":"172.21.15.188:6825/1230196525","cluster_addr":"172.21.15.188:6827/1230196525","heartbeat_back_addr":"172.21.15.188:6831/1230196525","heartbeat_front_addr":"172.21.15.188:6829/1230196525","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:46:26.803832+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:46:48.726253+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:47:09.744123+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:47:36.907588+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:47:57.696685+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:48:19.120806+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:48:45.313043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:49:06.686384+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:49:28.475078+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:49:54.984033+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:50:16.541259+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:50:38.221746+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/3892248290":"2023-07-21T02:44:53.458649+0000","172.21.15.7:0/2373058722":"2023-07-21T02:44:53.458649+0000","172.21.15.7:6801/2926310425":"2023-07-21T02:44:53.458649+0000","172.21.15.7:0/4222569264":"2023-07-21T02:44:25.804258+0000","172.21.15.7:0/1357065986":"2023-07-21T02:44:25.804258+0000","172.21.15.7:0/884368977":"2023-07-21T02:44:53.458649+0000","172.21.15.7:6801/1224472946":"2023-07-21T02:44:25.804258+0000","172.21.15.7:0/1459293748":"2023-07-21T02:44:25.804258+0000","172.21.15.7:6800/1224472946":"2023-07-21T02:44:25.804258+0000","172.21.15.7:6800/2926310425":"2023-07-21T02:44:53.458649+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T02:51:45.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.501+0000 7f3c6a1ff700 1 -- 172.21.15.7:0/3716987033 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f3c4c0a11a0 msgr2=0x7f3c4c0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:45.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.501+0000 7f3c6a1ff700 1 --2- 172.21.15.7:0/3716987033 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f3c4c0a11a0 0x7f3c4c0a3660 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3c641017c0 tx=0x7f3c54009260 comp rx=0 tx=0).stop 2023-07-20T02:51:45.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.501+0000 7f3c6a1ff700 1 -- 172.21.15.7:0/3716987033 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c64101960 msgr2=0x7f3c64072b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:45.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.501+0000 7f3c6a1ff700 1 --2- 172.21.15.7:0/3716987033 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c64101960 0x7f3c64072b60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3c5800e9a0 tx=0x7f3c5800ed60 comp rx=0 tx=0).stop 2023-07-20T02:51:45.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.501+0000 7f3c6a1ff700 1 -- 172.21.15.7:0/3716987033 shutdown_connections 2023-07-20T02:51:45.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.501+0000 7f3c6a1ff700 1 --2- 172.21.15.7:0/3716987033 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c64102300 0x7f3c640730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:45.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.501+0000 7f3c6a1ff700 1 --2- 172.21.15.7:0/3716987033 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3c64100770 0x7f3c64074520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:45.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.501+0000 7f3c6a1ff700 1 --2- 172.21.15.7:0/3716987033 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f3c4c0a11a0 0x7f3c4c0a3660 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:45.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.501+0000 7f3c6a1ff700 1 --2- 172.21.15.7:0/3716987033 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c64101960 0x7f3c64072b60 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:45.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.501+0000 7f3c6a1ff700 1 -- 172.21.15.7:0/3716987033 >> 172.21.15.7:0/3716987033 conn(0x7f3c640fbca0 msgr2=0x7f3c64104db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:45.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.501+0000 7f3c6a1ff700 1 -- 172.21.15.7:0/3716987033 shutdown_connections 2023-07-20T02:51:45.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.501+0000 7f3c6a1ff700 1 -- 172.21.15.7:0/3716987033 wait complete. 2023-07-20T02:51:45.533 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-20T02:47:12.055539+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-20T02:51:26.617961+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-20T02:51:28.081016+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-20T02:51:31.658088+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-20T02:51:45.534 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-20T02:51:45.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.729+0000 7fab24bff700 1 -- 172.21.15.7:0/515136232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab20073150 msgr2=0x7fab200735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:45.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.729+0000 7fab24bff700 1 --2- 172.21.15.7:0/515136232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab20073150 0x7fab200735d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fab14009990 tx=0x7fab14009ca0 comp rx=0 tx=0).stop 2023-07-20T02:51:45.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.729+0000 7fab24bff700 1 -- 172.21.15.7:0/515136232 shutdown_connections 2023-07-20T02:51:45.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.729+0000 7fab24bff700 1 --2- 172.21.15.7:0/515136232 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab20108650 0x7fab2010aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:45.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.729+0000 7fab24bff700 1 --2- 172.21.15.7:0/515136232 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab20074730 0x7fab20072b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:45.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.729+0000 7fab24bff700 1 --2- 172.21.15.7:0/515136232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab20073150 0x7fab200735d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:45.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.729+0000 7fab24bff700 1 -- 172.21.15.7:0/515136232 >> 172.21.15.7:0/515136232 conn(0x7fab200fbcb0 msgr2=0x7fab200fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:45.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.729+0000 7fab24bff700 1 -- 172.21.15.7:0/515136232 shutdown_connections 2023-07-20T02:51:45.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.730+0000 7fab24bff700 1 -- 172.21.15.7:0/515136232 wait complete. 2023-07-20T02:51:45.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.730+0000 7fab24bff700 1 Processor -- start 2023-07-20T02:51:45.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.731+0000 7fab24bff700 1 -- start start 2023-07-20T02:51:45.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.731+0000 7fab24bff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab20073150 0x7fab20198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:45.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.731+0000 7fab24bff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab20074730 0x7fab20198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:45.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.731+0000 7fab24bff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab20108650 0x7fab2019fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:45.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.731+0000 7fab24bff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab201a0200 con 0x7fab20074730 2023-07-20T02:51:45.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.731+0000 7fab24bff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab201a0340 con 0x7fab20108650 2023-07-20T02:51:45.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.731+0000 7fab24bff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab201a04d0 con 0x7fab20073150 2023-07-20T02:51:45.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.731+0000 7fab1dd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab20074730 0x7fab20198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:45.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.731+0000 7fab1e59c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab20073150 0x7fab20198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:45.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.731+0000 7fab1dd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab20074730 0x7fab20198bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48498/0 (socket says 172.21.15.7:48498) 2023-07-20T02:51:45.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.731+0000 7fab1dd9b700 1 -- 172.21.15.7:0/3719068450 learned_addr learned my addr 172.21.15.7:0/3719068450 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:45.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.731+0000 7fab1ed9d700 1 --2- 172.21.15.7:0/3719068450 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab20108650 0x7fab2019fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:45.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.732+0000 7fab1dd9b700 1 -- 172.21.15.7:0/3719068450 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab20073150 msgr2=0x7fab20198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:45.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.732+0000 7fab1dd9b700 1 --2- 172.21.15.7:0/3719068450 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab20073150 0x7fab20198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:45.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.732+0000 7fab1dd9b700 1 -- 172.21.15.7:0/3719068450 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab20108650 msgr2=0x7fab2019fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:45.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.732+0000 7fab1dd9b700 1 --2- 172.21.15.7:0/3719068450 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab20108650 0x7fab2019fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:45.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.732+0000 7fab1dd9b700 1 -- 172.21.15.7:0/3719068450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab140095f0 con 0x7fab20074730 2023-07-20T02:51:45.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.732+0000 7fab1ed9d700 1 --2- 172.21.15.7:0/3719068450 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab20108650 0x7fab2019fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T02:51:45.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.732+0000 7fab1dd9b700 1 --2- 172.21.15.7:0/3719068450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab20074730 0x7fab20198bc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fab14004740 tx=0x7fab14009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:45.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.732+0000 7fab0f7fe700 1 -- 172.21.15.7:0/3719068450 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab1401d070 con 0x7fab20074730 2023-07-20T02:51:45.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.732+0000 7fab0f7fe700 1 -- 172.21.15.7:0/3719068450 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fab14004830 con 0x7fab20074730 2023-07-20T02:51:45.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.732+0000 7fab24bff700 1 -- 172.21.15.7:0/3719068450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab201a0750 con 0x7fab20074730 2023-07-20T02:51:45.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.732+0000 7fab0f7fe700 1 -- 172.21.15.7:0/3719068450 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab14005450 con 0x7fab20074730 2023-07-20T02:51:45.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.732+0000 7fab24bff700 1 -- 172.21.15.7:0/3719068450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab201a0b10 con 0x7fab20074730 2023-07-20T02:51:45.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.733+0000 7fab24bff700 1 -- 172.21.15.7:0/3719068450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab20106d50 con 0x7fab20074730 2023-07-20T02:51:45.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.735+0000 7fab0f7fe700 1 -- 172.21.15.7:0/3719068450 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fab1400fb00 con 0x7fab20074730 2023-07-20T02:51:45.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.737+0000 7fab0f7fe700 1 --2- 172.21.15.7:0/3719068450 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fab080a1150 0x7fab080a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:45.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.737+0000 7fab0f7fe700 1 -- 172.21.15.7:0/3719068450 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fab140cdd20 con 0x7fab20074730 2023-07-20T02:51:45.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.737+0000 7fab1e59c700 1 --2- 172.21.15.7:0/3719068450 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fab080a1150 0x7fab080a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:45.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.738+0000 7fab1e59c700 1 --2- 172.21.15.7:0/3719068450 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fab080a1150 0x7fab080a3610 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fab10009c60 tx=0x7fab10009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:45.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.741+0000 7fab0f7fe700 1 -- 172.21.15.7:0/3719068450 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fab1405a780 con 0x7fab20074730 2023-07-20T02:51:45.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:45 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/3716987033' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:51:45.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:45 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/3716987033' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:51:45.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.948+0000 7fab24bff700 1 -- 172.21.15.7:0/3719068450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fab20061420 con 0x7fab20074730 2023-07-20T02:51:45.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.949+0000 7fab0f7fe700 1 -- 172.21.15.7:0/3719068450 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7fab140972b0 con 0x7fab20074730 2023-07-20T02:51:45.963 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2023-07-20T02:51:45.963 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:45 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/3716987033' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:51:45.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.951+0000 7fab24bff700 1 -- 172.21.15.7:0/3719068450 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fab080a1150 msgr2=0x7fab080a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:45.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.951+0000 7fab24bff700 1 --2- 172.21.15.7:0/3719068450 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fab080a1150 0x7fab080a3610 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fab10009c60 tx=0x7fab10009260 comp rx=0 tx=0).stop 2023-07-20T02:51:45.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.951+0000 7fab24bff700 1 -- 172.21.15.7:0/3719068450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab20074730 msgr2=0x7fab20198bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:45.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.951+0000 7fab24bff700 1 --2- 172.21.15.7:0/3719068450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab20074730 0x7fab20198bc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fab14004740 tx=0x7fab14009580 comp rx=0 tx=0).stop 2023-07-20T02:51:45.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.952+0000 7fab24bff700 1 -- 172.21.15.7:0/3719068450 shutdown_connections 2023-07-20T02:51:45.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.952+0000 7fab24bff700 1 --2- 172.21.15.7:0/3719068450 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fab20108650 0x7fab2019fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:45.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.952+0000 7fab24bff700 1 --2- 172.21.15.7:0/3719068450 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab20073150 0x7fab20198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:45.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.952+0000 7fab24bff700 1 --2- 172.21.15.7:0/3719068450 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fab080a1150 0x7fab080a3610 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:45.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.952+0000 7fab24bff700 1 --2- 172.21.15.7:0/3719068450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab20074730 0x7fab20198bc0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:45.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.952+0000 7fab24bff700 1 -- 172.21.15.7:0/3719068450 >> 172.21.15.7:0/3719068450 conn(0x7fab200fbcb0 msgr2=0x7fab200fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:45.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.952+0000 7fab24bff700 1 -- 172.21.15.7:0/3719068450 shutdown_connections 2023-07-20T02:51:45.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:45.952+0000 7fab24bff700 1 -- 172.21.15.7:0/3719068450 wait complete. 2023-07-20T02:51:45.983 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-20T02:51:46.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.177+0000 7f347a3ff700 1 -- 172.21.15.7:0/736128623 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3474101970 msgr2=0x7f3474101dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:46.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.177+0000 7f347a3ff700 1 --2- 172.21.15.7:0/736128623 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3474101970 0x7f3474101dd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f34680099c0 tx=0x7f3468009cd0 comp rx=0 tx=0).stop 2023-07-20T02:51:46.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.177+0000 7f347a3ff700 1 -- 172.21.15.7:0/736128623 shutdown_connections 2023-07-20T02:51:46.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.177+0000 7f347a3ff700 1 --2- 172.21.15.7:0/736128623 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3474102310 0x7f3474108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:46.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.177+0000 7f347a3ff700 1 --2- 172.21.15.7:0/736128623 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3474100780 0x7f3474100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:46.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.177+0000 7f347a3ff700 1 --2- 172.21.15.7:0/736128623 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3474101970 0x7f3474101dd0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:46.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.177+0000 7f347a3ff700 1 -- 172.21.15.7:0/736128623 >> 172.21.15.7:0/736128623 conn(0x7f34740fbcf0 msgr2=0x7f34740fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:46.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.177+0000 7f347a3ff700 1 -- 172.21.15.7:0/736128623 shutdown_connections 2023-07-20T02:51:46.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.177+0000 7f347a3ff700 1 -- 172.21.15.7:0/736128623 wait complete. 2023-07-20T02:51:46.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.178+0000 7f347a3ff700 1 Processor -- start 2023-07-20T02:51:46.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.178+0000 7f347a3ff700 1 -- start start 2023-07-20T02:51:46.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.178+0000 7f347a3ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3474100780 0x7f34741964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:46.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.178+0000 7f347a3ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3474101970 0x7f34741969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:46.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.178+0000 7f347a3ff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3474102310 0x7f347419da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:46.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.178+0000 7f347a3ff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347419e030 con 0x7f3474100780 2023-07-20T02:51:46.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.178+0000 7f347a3ff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347419e170 con 0x7f3474102310 2023-07-20T02:51:46.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.178+0000 7f347a3ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347419e300 con 0x7f3474101970 2023-07-20T02:51:46.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.179+0000 7f3473fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3474100780 0x7f34741964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:46.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.179+0000 7f3473fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3474100780 0x7f34741964b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48512/0 (socket says 172.21.15.7:48512) 2023-07-20T02:51:46.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.179+0000 7f3473fff700 1 -- 172.21.15.7:0/3394902229 learned_addr learned my addr 172.21.15.7:0/3394902229 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:46.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.179+0000 7f34737fe700 1 --2- 172.21.15.7:0/3394902229 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3474101970 0x7f34741969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:46.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.179+0000 7f347899c700 1 --2- 172.21.15.7:0/3394902229 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3474102310 0x7f347419da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:46.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.179+0000 7f3473fff700 1 -- 172.21.15.7:0/3394902229 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3474101970 msgr2=0x7f34741969f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:46.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.179+0000 7f3473fff700 1 --2- 172.21.15.7:0/3394902229 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3474101970 0x7f34741969f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:46.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.179+0000 7f3473fff700 1 -- 172.21.15.7:0/3394902229 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3474102310 msgr2=0x7f347419da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:46.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.179+0000 7f3473fff700 1 --2- 172.21.15.7:0/3394902229 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3474102310 0x7f347419da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:46.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.179+0000 7f3473fff700 1 -- 172.21.15.7:0/3394902229 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34680095f0 con 0x7f3474100780 2023-07-20T02:51:46.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.179+0000 7f3473fff700 1 --2- 172.21.15.7:0/3394902229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3474100780 0x7f34741964b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f346400e970 tx=0x7f346400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:46.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.179+0000 7f34717fa700 1 -- 172.21.15.7:0/3394902229 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3464009790 con 0x7f3474100780 2023-07-20T02:51:46.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.180+0000 7f34717fa700 1 -- 172.21.15.7:0/3394902229 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f346400c7e0 con 0x7f3474100780 2023-07-20T02:51:46.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.180+0000 7f347a3ff700 1 -- 172.21.15.7:0/3394902229 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f347419e5e0 con 0x7f3474100780 2023-07-20T02:51:46.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.180+0000 7f34717fa700 1 -- 172.21.15.7:0/3394902229 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3464003c70 con 0x7f3474100780 2023-07-20T02:51:46.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.180+0000 7f347a3ff700 1 -- 172.21.15.7:0/3394902229 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f347419eb40 con 0x7f3474100780 2023-07-20T02:51:46.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.181+0000 7f347a3ff700 1 -- 172.21.15.7:0/3394902229 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f347410bff0 con 0x7f3474100780 2023-07-20T02:51:46.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.184+0000 7f34717fa700 1 -- 172.21.15.7:0/3394902229 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f346400cc50 con 0x7f3474100780 2023-07-20T02:51:46.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.186+0000 7f34717fa700 1 --2- 172.21.15.7:0/3394902229 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f345c0a11d0 0x7f345c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:46.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.186+0000 7f34717fa700 1 -- 172.21.15.7:0/3394902229 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f3464014070 con 0x7f3474100780 2023-07-20T02:51:46.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.186+0000 7f34737fe700 1 --2- 172.21.15.7:0/3394902229 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f345c0a11d0 0x7f345c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:46.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.187+0000 7f34737fe700 1 --2- 172.21.15.7:0/3394902229 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f345c0a11d0 0x7f345c0a3690 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f346800b580 tx=0x7f3468005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:46.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.188+0000 7f34717fa700 1 -- 172.21.15.7:0/3394902229 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f34640962c0 con 0x7f3474100780 2023-07-20T02:51:46.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.400+0000 7f347a3ff700 1 -- 172.21.15.7:0/3394902229 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f34741076b0 con 0x7f3474100780 2023-07-20T02:51:46.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.400+0000 7f34717fa700 1 -- 172.21.15.7:0/3394902229 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f34640998e0 con 0x7f3474100780 2023-07-20T02:51:46.414 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2023-07-20T02:51:46.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.402+0000 7f347a3ff700 1 -- 172.21.15.7:0/3394902229 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f345c0a11d0 msgr2=0x7f345c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:46.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.402+0000 7f347a3ff700 1 --2- 172.21.15.7:0/3394902229 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f345c0a11d0 0x7f345c0a3690 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f346800b580 tx=0x7f3468005fb0 comp rx=0 tx=0).stop 2023-07-20T02:51:46.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.402+0000 7f347a3ff700 1 -- 172.21.15.7:0/3394902229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3474100780 msgr2=0x7f34741964b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:46.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.402+0000 7f347a3ff700 1 --2- 172.21.15.7:0/3394902229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3474100780 0x7f34741964b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f346400e970 tx=0x7f346400ec80 comp rx=0 tx=0).stop 2023-07-20T02:51:46.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.402+0000 7f347a3ff700 1 -- 172.21.15.7:0/3394902229 shutdown_connections 2023-07-20T02:51:46.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.402+0000 7f347a3ff700 1 --2- 172.21.15.7:0/3394902229 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f3474102310 0x7f347419da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:46.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.402+0000 7f347a3ff700 1 --2- 172.21.15.7:0/3394902229 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f345c0a11d0 0x7f345c0a3690 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:46.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.402+0000 7f347a3ff700 1 --2- 172.21.15.7:0/3394902229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3474100780 0x7f34741964b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:46.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.402+0000 7f347a3ff700 1 --2- 172.21.15.7:0/3394902229 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3474101970 0x7f34741969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:46.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.403+0000 7f347a3ff700 1 -- 172.21.15.7:0/3394902229 >> 172.21.15.7:0/3394902229 conn(0x7f34740fbcf0 msgr2=0x7f34740fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:46.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.403+0000 7f347a3ff700 1 -- 172.21.15.7:0/3394902229 shutdown_connections 2023-07-20T02:51:46.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.403+0000 7f347a3ff700 1 -- 172.21.15.7:0/3394902229 wait complete. 2023-07-20T02:51:46.434 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-20T02:51:46.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.628+0000 7fdd4cbff700 1 -- 172.21.15.7:0/2714307518 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd48074750 msgr2=0x7fdd48072ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:46.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.628+0000 7fdd4cbff700 1 --2- 172.21.15.7:0/2714307518 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd48074750 0x7fdd48072ba0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdd38009990 tx=0x7fdd38009ca0 comp rx=0 tx=0).stop 2023-07-20T02:51:46.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.628+0000 7fdd4cbff700 1 -- 172.21.15.7:0/2714307518 shutdown_connections 2023-07-20T02:51:46.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.628+0000 7fdd4cbff700 1 --2- 172.21.15.7:0/2714307518 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd48108630 0x7fdd4810aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:46.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.628+0000 7fdd4cbff700 1 --2- 172.21.15.7:0/2714307518 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd48074750 0x7fdd48072ba0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:46.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.628+0000 7fdd4cbff700 1 --2- 172.21.15.7:0/2714307518 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd48073170 0x7fdd480735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:46.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.628+0000 7fdd4cbff700 1 -- 172.21.15.7:0/2714307518 >> 172.21.15.7:0/2714307518 conn(0x7fdd480fbcb0 msgr2=0x7fdd480fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:46.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.628+0000 7fdd4cbff700 1 -- 172.21.15.7:0/2714307518 shutdown_connections 2023-07-20T02:51:46.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.628+0000 7fdd4cbff700 1 -- 172.21.15.7:0/2714307518 wait complete. 2023-07-20T02:51:46.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.629+0000 7fdd4cbff700 1 Processor -- start 2023-07-20T02:51:46.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.629+0000 7fdd4cbff700 1 -- start start 2023-07-20T02:51:46.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.630+0000 7fdd4cbff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd48073170 0x7fdd48198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:46.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.630+0000 7fdd4cbff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd48074750 0x7fdd48198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:46.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.630+0000 7fdd4cbff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd48108630 0x7fdd4819fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:46.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.630+0000 7fdd4cbff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd481a0200 con 0x7fdd48108630 2023-07-20T02:51:46.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.630+0000 7fdd4cbff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd481a0340 con 0x7fdd48073170 2023-07-20T02:51:46.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.630+0000 7fdd4cbff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd481a04d0 con 0x7fdd48074750 2023-07-20T02:51:46.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.630+0000 7fdd46d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd48108630 0x7fdd4819fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:46.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.630+0000 7fdd4659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd48073170 0x7fdd48198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:46.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.630+0000 7fdd4659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd48073170 0x7fdd48198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.7:44526/0 (socket says 172.21.15.7:44526) 2023-07-20T02:51:46.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.630+0000 7fdd46d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd48108630 0x7fdd4819fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48528/0 (socket says 172.21.15.7:48528) 2023-07-20T02:51:46.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.630+0000 7fdd4659c700 1 -- 172.21.15.7:0/2975453906 learned_addr learned my addr 172.21.15.7:0/2975453906 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:46.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.630+0000 7fdd45d9b700 1 --2- 172.21.15.7:0/2975453906 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd48074750 0x7fdd48198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:46.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.630+0000 7fdd46d9d700 1 -- 172.21.15.7:0/2975453906 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd48074750 msgr2=0x7fdd48198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:46.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.630+0000 7fdd46d9d700 1 --2- 172.21.15.7:0/2975453906 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd48074750 0x7fdd48198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:46.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.630+0000 7fdd46d9d700 1 -- 172.21.15.7:0/2975453906 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd48073170 msgr2=0x7fdd48198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:46.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.630+0000 7fdd46d9d700 1 --2- 172.21.15.7:0/2975453906 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd48073170 0x7fdd48198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:46.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.630+0000 7fdd46d9d700 1 -- 172.21.15.7:0/2975453906 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd3c009520 con 0x7fdd48108630 2023-07-20T02:51:46.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.631+0000 7fdd46d9d700 1 --2- 172.21.15.7:0/2975453906 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd48108630 0x7fdd4819fc30 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdd4000efc0 tx=0x7fdd4000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:46.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.631+0000 7fdd377fe700 1 -- 172.21.15.7:0/2975453906 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd40018070 con 0x7fdd48108630 2023-07-20T02:51:46.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.631+0000 7fdd377fe700 1 -- 172.21.15.7:0/2975453906 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd4000f040 con 0x7fdd48108630 2023-07-20T02:51:46.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.631+0000 7fdd4cbff700 1 -- 172.21.15.7:0/2975453906 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd380095f0 con 0x7fdd48108630 2023-07-20T02:51:46.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.631+0000 7fdd377fe700 1 -- 172.21.15.7:0/2975453906 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4000cd40 con 0x7fdd48108630 2023-07-20T02:51:46.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.631+0000 7fdd4cbff700 1 -- 172.21.15.7:0/2975453906 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd481a0ad0 con 0x7fdd48108630 2023-07-20T02:51:46.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.632+0000 7fdd4cbff700 1 -- 172.21.15.7:0/2975453906 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd48106d30 con 0x7fdd48108630 2023-07-20T02:51:46.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.634+0000 7fdd377fe700 1 -- 172.21.15.7:0/2975453906 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fdd40003a00 con 0x7fdd48108630 2023-07-20T02:51:46.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.635+0000 7fdd377fe700 1 --2- 172.21.15.7:0/2975453906 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fdd300a11d0 0x7fdd300a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:46.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.636+0000 7fdd377fe700 1 -- 172.21.15.7:0/2975453906 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fdd400cc690 con 0x7fdd48108630 2023-07-20T02:51:46.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.636+0000 7fdd4659c700 1 --2- 172.21.15.7:0/2975453906 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fdd300a11d0 0x7fdd300a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:46.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.636+0000 7fdd4659c700 1 --2- 172.21.15.7:0/2975453906 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fdd300a11d0 0x7fdd300a3690 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fdd3800b2e0 tx=0x7fdd38005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:46.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.639+0000 7fdd377fe700 1 -- 172.21.15.7:0/2975453906 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fdd40095c20 con 0x7fdd48108630 2023-07-20T02:51:46.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:46 smithi007 ceph-mon[126353]: pgmap v283: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-20T02:51:46.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:46 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/3719068450' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T02:51:46.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:46 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/3394902229' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-20T02:51:46.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.849+0000 7fdd4cbff700 1 -- 172.21.15.7:0/2975453906 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdd48061380 con 0x7fdd48108630 2023-07-20T02:51:46.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.850+0000 7fdd377fe700 1 -- 172.21.15.7:0/2975453906 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdd40099240 con 0x7fdd48108630 2023-07-20T02:51:46.864 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2023-07-20T02:51:46.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.852+0000 7fdd4cbff700 1 -- 172.21.15.7:0/2975453906 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fdd300a11d0 msgr2=0x7fdd300a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:46.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.852+0000 7fdd4cbff700 1 --2- 172.21.15.7:0/2975453906 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fdd300a11d0 0x7fdd300a3690 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fdd3800b2e0 tx=0x7fdd38005fb0 comp rx=0 tx=0).stop 2023-07-20T02:51:46.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.852+0000 7fdd4cbff700 1 -- 172.21.15.7:0/2975453906 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd48108630 msgr2=0x7fdd4819fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:46.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.852+0000 7fdd4cbff700 1 --2- 172.21.15.7:0/2975453906 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd48108630 0x7fdd4819fc30 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdd4000efc0 tx=0x7fdd4000c3a0 comp rx=0 tx=0).stop 2023-07-20T02:51:46.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.852+0000 7fdd4cbff700 1 -- 172.21.15.7:0/2975453906 shutdown_connections 2023-07-20T02:51:46.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.852+0000 7fdd4cbff700 1 --2- 172.21.15.7:0/2975453906 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fdd300a11d0 0x7fdd300a3690 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:46.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.852+0000 7fdd4cbff700 1 --2- 172.21.15.7:0/2975453906 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd48108630 0x7fdd4819fc30 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:46.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.852+0000 7fdd4cbff700 1 --2- 172.21.15.7:0/2975453906 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fdd48073170 0x7fdd48198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:46.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.852+0000 7fdd4cbff700 1 --2- 172.21.15.7:0/2975453906 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd48074750 0x7fdd48198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:46.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.852+0000 7fdd4cbff700 1 -- 172.21.15.7:0/2975453906 >> 172.21.15.7:0/2975453906 conn(0x7fdd480fbcb0 msgr2=0x7fdd480fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:46.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.852+0000 7fdd4cbff700 1 -- 172.21.15.7:0/2975453906 shutdown_connections 2023-07-20T02:51:46.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:46.852+0000 7fdd4cbff700 1 -- 172.21.15.7:0/2975453906 wait complete. 2023-07-20T02:51:46.884 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-20T02:51:46.890 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:46 smithi158 ceph-mon[128925]: pgmap v283: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-20T02:51:46.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:46 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/3719068450' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T02:51:46.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:46 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/3394902229' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-20T02:51:46.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:46 smithi188 ceph-mon[129553]: pgmap v283: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-20T02:51:46.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:46 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/3719068450' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T02:51:46.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:46 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/3394902229' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-20T02:51:47.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.077+0000 7f61673ff700 1 -- 172.21.15.7:0/3834877037 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6160100760 msgr2=0x7f6160100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:47.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.077+0000 7f61673ff700 1 --2- 172.21.15.7:0/3834877037 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6160100760 0x7f6160100b80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f615c0099c0 tx=0x7f615c009cd0 comp rx=0 tx=0).stop 2023-07-20T02:51:47.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.077+0000 7f61673ff700 1 -- 172.21.15.7:0/3834877037 shutdown_connections 2023-07-20T02:51:47.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.077+0000 7f61673ff700 1 --2- 172.21.15.7:0/3834877037 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61601022f0 0x7f6160108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:47.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.077+0000 7f61673ff700 1 --2- 172.21.15.7:0/3834877037 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6160100760 0x7f6160100b80 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:47.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.077+0000 7f61673ff700 1 --2- 172.21.15.7:0/3834877037 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6160101950 0x7f6160101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:47.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.077+0000 7f61673ff700 1 -- 172.21.15.7:0/3834877037 >> 172.21.15.7:0/3834877037 conn(0x7f61600fbcb0 msgr2=0x7f61600fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:47.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.077+0000 7f61673ff700 1 -- 172.21.15.7:0/3834877037 shutdown_connections 2023-07-20T02:51:47.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.077+0000 7f61673ff700 1 -- 172.21.15.7:0/3834877037 wait complete. 2023-07-20T02:51:47.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.078+0000 7f61673ff700 1 Processor -- start 2023-07-20T02:51:47.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.079+0000 7f61673ff700 1 -- start start 2023-07-20T02:51:47.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.079+0000 7f61673ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6160100760 0x7f61601986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:47.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.079+0000 7f61673ff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6160101950 0x7f6160198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:47.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.079+0000 7f61673ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61601022f0 0x7f616019fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:47.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.079+0000 7f61673ff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61601a0260 con 0x7f6160100760 2023-07-20T02:51:47.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.079+0000 7f61673ff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61601a03a0 con 0x7f6160101950 2023-07-20T02:51:47.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.079+0000 7f61673ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61601a0530 con 0x7f61601022f0 2023-07-20T02:51:47.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.079+0000 7f616519b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6160100760 0x7f61601986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:47.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.079+0000 7f616519b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6160100760 0x7f61601986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48544/0 (socket says 172.21.15.7:48544) 2023-07-20T02:51:47.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.079+0000 7f616519b700 1 -- 172.21.15.7:0/1212611308 learned_addr learned my addr 172.21.15.7:0/1212611308 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:47.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.079+0000 7f616599c700 1 --2- 172.21.15.7:0/1212611308 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61601022f0 0x7f616019fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:47.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.079+0000 7f616499a700 1 --2- 172.21.15.7:0/1212611308 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6160101950 0x7f6160198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:47.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.080+0000 7f616519b700 1 -- 172.21.15.7:0/1212611308 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61601022f0 msgr2=0x7f616019fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:47.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.080+0000 7f616519b700 1 --2- 172.21.15.7:0/1212611308 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61601022f0 0x7f616019fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:47.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.080+0000 7f616519b700 1 -- 172.21.15.7:0/1212611308 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6160101950 msgr2=0x7f6160198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:47.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.080+0000 7f616519b700 1 --2- 172.21.15.7:0/1212611308 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6160101950 0x7f6160198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:47.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.080+0000 7f616519b700 1 -- 172.21.15.7:0/1212611308 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f615c0095f0 con 0x7f6160100760 2023-07-20T02:51:47.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.080+0000 7f616519b700 1 --2- 172.21.15.7:0/1212611308 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6160100760 0x7f61601986e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f615c009fd0 tx=0x7f615c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:47.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.080+0000 7f61567fc700 1 -- 172.21.15.7:0/1212611308 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c01d070 con 0x7f6160100760 2023-07-20T02:51:47.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.080+0000 7f61567fc700 1 -- 172.21.15.7:0/1212611308 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f615c005030 con 0x7f6160100760 2023-07-20T02:51:47.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.080+0000 7f61673ff700 1 -- 172.21.15.7:0/1212611308 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61601a07b0 con 0x7f6160100760 2023-07-20T02:51:47.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.080+0000 7f61567fc700 1 -- 172.21.15.7:0/1212611308 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c004a90 con 0x7f6160100760 2023-07-20T02:51:47.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.080+0000 7f61673ff700 1 -- 172.21.15.7:0/1212611308 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61601a0bd0 con 0x7f6160100760 2023-07-20T02:51:47.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.081+0000 7f61673ff700 1 -- 172.21.15.7:0/1212611308 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6160192d60 con 0x7f6160100760 2023-07-20T02:51:47.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.083+0000 7f61567fc700 1 -- 172.21.15.7:0/1212611308 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f615c0051a0 con 0x7f6160100760 2023-07-20T02:51:47.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.084+0000 7f61567fc700 1 --2- 172.21.15.7:0/1212611308 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f614c0a1150 0x7f614c0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:47.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.085+0000 7f61567fc700 1 -- 172.21.15.7:0/1212611308 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f615c0cd8c0 con 0x7f6160100760 2023-07-20T02:51:47.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.085+0000 7f616499a700 1 --2- 172.21.15.7:0/1212611308 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f614c0a1150 0x7f614c0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:47.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.085+0000 7f616499a700 1 --2- 172.21.15.7:0/1212611308 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f614c0a1150 0x7f614c0a3610 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6150009c60 tx=0x7f6150009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:47.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.088+0000 7f61567fc700 1 -- 172.21.15.7:0/1212611308 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f615c096e50 con 0x7f6160100760 2023-07-20T02:51:47.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.298+0000 7f61673ff700 1 -- 172.21.15.7:0/1212611308 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f6160061380 con 0x7f6160100760 2023-07-20T02:51:47.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.298+0000 7f61567fc700 1 -- 172.21.15.7:0/1212611308 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v87) v1 ==== 103+0+11 (secure 0 0 0) 0x7f615c09a470 con 0x7f6160100760 2023-07-20T02:51:47.313 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2023-07-20T02:51:47.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.301+0000 7f61673ff700 1 -- 172.21.15.7:0/1212611308 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f614c0a1150 msgr2=0x7f614c0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:47.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.301+0000 7f61673ff700 1 --2- 172.21.15.7:0/1212611308 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f614c0a1150 0x7f614c0a3610 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6150009c60 tx=0x7f6150009260 comp rx=0 tx=0).stop 2023-07-20T02:51:47.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.301+0000 7f61673ff700 1 -- 172.21.15.7:0/1212611308 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6160100760 msgr2=0x7f61601986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:47.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.301+0000 7f61673ff700 1 --2- 172.21.15.7:0/1212611308 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6160100760 0x7f61601986e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f615c009fd0 tx=0x7f615c009580 comp rx=0 tx=0).stop 2023-07-20T02:51:47.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.301+0000 7f61673ff700 1 -- 172.21.15.7:0/1212611308 shutdown_connections 2023-07-20T02:51:47.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.301+0000 7f61673ff700 1 --2- 172.21.15.7:0/1212611308 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61601022f0 0x7f616019fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:47.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.301+0000 7f61673ff700 1 --2- 172.21.15.7:0/1212611308 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f614c0a1150 0x7f614c0a3610 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:47.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.301+0000 7f61673ff700 1 --2- 172.21.15.7:0/1212611308 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6160100760 0x7f61601986e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:47.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.301+0000 7f61673ff700 1 --2- 172.21.15.7:0/1212611308 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6160101950 0x7f6160198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:47.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.301+0000 7f61673ff700 1 -- 172.21.15.7:0/1212611308 >> 172.21.15.7:0/1212611308 conn(0x7f61600fbcb0 msgr2=0x7f6160107330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:47.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.301+0000 7f61673ff700 1 -- 172.21.15.7:0/1212611308 shutdown_connections 2023-07-20T02:51:47.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.301+0000 7f61673ff700 1 -- 172.21.15.7:0/1212611308 wait complete. 2023-07-20T02:51:47.334 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-07-20T02:51:47.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.526+0000 7fb54cfff700 1 -- 172.21.15.7:0/1173834445 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb548101950 msgr2=0x7fb548101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:47.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.526+0000 7fb54cfff700 1 --2- 172.21.15.7:0/1173834445 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb548101950 0x7fb548101db0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb53c009920 tx=0x7fb53c009c30 comp rx=0 tx=0).stop 2023-07-20T02:51:47.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.526+0000 7fb54cfff700 1 -- 172.21.15.7:0/1173834445 shutdown_connections 2023-07-20T02:51:47.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.526+0000 7fb54cfff700 1 --2- 172.21.15.7:0/1173834445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb5481022f0 0x7fb548108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:47.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.526+0000 7fb54cfff700 1 --2- 172.21.15.7:0/1173834445 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb548100760 0x7fb548100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:47.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.526+0000 7fb54cfff700 1 --2- 172.21.15.7:0/1173834445 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb548101950 0x7fb548101db0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:47.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.526+0000 7fb54cfff700 1 -- 172.21.15.7:0/1173834445 >> 172.21.15.7:0/1173834445 conn(0x7fb5480fbcb0 msgr2=0x7fb5480fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:47.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.526+0000 7fb54cfff700 1 -- 172.21.15.7:0/1173834445 shutdown_connections 2023-07-20T02:51:47.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.526+0000 7fb54cfff700 1 -- 172.21.15.7:0/1173834445 wait complete. 2023-07-20T02:51:47.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.527+0000 7fb54cfff700 1 Processor -- start 2023-07-20T02:51:47.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.527+0000 7fb54cfff700 1 -- start start 2023-07-20T02:51:47.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.527+0000 7fb54cfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb548100760 0x7fb548074520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:47.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.527+0000 7fb54cfff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb548101950 0x7fb548072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:47.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.527+0000 7fb54cfff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb5481022f0 0x7fb5480730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:47.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.527+0000 7fb54cfff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb548073840 con 0x7fb548100760 2023-07-20T02:51:47.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.527+0000 7fb54cfff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5480739b0 con 0x7fb548101950 2023-07-20T02:51:47.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.527+0000 7fb54cfff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5481a0520 con 0x7fb5481022f0 2023-07-20T02:51:47.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.528+0000 7fb54659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb548100760 0x7fb548074520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:47.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.528+0000 7fb54659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb548100760 0x7fb548074520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48560/0 (socket says 172.21.15.7:48560) 2023-07-20T02:51:47.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.528+0000 7fb54659c700 1 -- 172.21.15.7:0/466873929 learned_addr learned my addr 172.21.15.7:0/466873929 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:47.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.528+0000 7fb546d9d700 1 --2- 172.21.15.7:0/466873929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb5481022f0 0x7fb5480730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:47.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.528+0000 7fb545d9b700 1 --2- 172.21.15.7:0/466873929 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb548101950 0x7fb548072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:47.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.528+0000 7fb54659c700 1 -- 172.21.15.7:0/466873929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb5481022f0 msgr2=0x7fb5480730a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:47.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.528+0000 7fb54659c700 1 --2- 172.21.15.7:0/466873929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb5481022f0 0x7fb5480730a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:47.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.528+0000 7fb54659c700 1 -- 172.21.15.7:0/466873929 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb548101950 msgr2=0x7fb548072b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:47.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.528+0000 7fb54659c700 1 --2- 172.21.15.7:0/466873929 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb548101950 0x7fb548072b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:47.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.528+0000 7fb54659c700 1 -- 172.21.15.7:0/466873929 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb53c009520 con 0x7fb548100760 2023-07-20T02:51:47.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.528+0000 7fb54659c700 1 --2- 172.21.15.7:0/466873929 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb548100760 0x7fb548074520 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb53800e9f0 tx=0x7fb53800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:47.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.529+0000 7fb5377fe700 1 -- 172.21.15.7:0/466873929 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb538004220 con 0x7fb548100760 2023-07-20T02:51:47.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.529+0000 7fb5377fe700 1 -- 172.21.15.7:0/466873929 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb538004380 con 0x7fb548100760 2023-07-20T02:51:47.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.529+0000 7fb54cfff700 1 -- 172.21.15.7:0/466873929 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5481a0800 con 0x7fb548100760 2023-07-20T02:51:47.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.529+0000 7fb5377fe700 1 -- 172.21.15.7:0/466873929 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb538003be0 con 0x7fb548100760 2023-07-20T02:51:47.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.529+0000 7fb54cfff700 1 -- 172.21.15.7:0/466873929 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5481a0d10 con 0x7fb548100760 2023-07-20T02:51:47.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.530+0000 7fb54cfff700 1 -- 172.21.15.7:0/466873929 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb54810bf80 con 0x7fb548100760 2023-07-20T02:51:47.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.532+0000 7fb5377fe700 1 -- 172.21.15.7:0/466873929 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fb538003d40 con 0x7fb548100760 2023-07-20T02:51:47.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.533+0000 7fb5377fe700 1 --2- 172.21.15.7:0/466873929 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fb5300a1150 0x7fb5300a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:47.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.533+0000 7fb5377fe700 1 -- 172.21.15.7:0/466873929 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fb538014070 con 0x7fb548100760 2023-07-20T02:51:47.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.533+0000 7fb545d9b700 1 --2- 172.21.15.7:0/466873929 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fb5300a1150 0x7fb5300a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:47.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.534+0000 7fb545d9b700 1 --2- 172.21.15.7:0/466873929 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fb5300a1150 0x7fb5300a3610 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb53c00fa40 tx=0x7fb53c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:47.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.536+0000 7fb5377fe700 1 -- 172.21.15.7:0/466873929 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb538096260 con 0x7fb548100760 2023-07-20T02:51:47.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.760+0000 7fb54cfff700 1 -- 172.21.15.7:0/466873929 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb5481076f0 con 0x7fb548100760 2023-07-20T02:51:47.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.761+0000 7fb5377fe700 1 -- 172.21.15.7:0/466873929 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v87) v1 ==== 81+0+581 (secure 0 0 0) 0x7fb538099880 con 0x7fb548100760 2023-07-20T02:51:47.775 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/3892248290 2023-07-21T02:44:53.458649+0000 2023-07-20T02:51:47.775 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/2373058722 2023-07-21T02:44:53.458649+0000 2023-07-20T02:51:47.775 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6801/2926310425 2023-07-21T02:44:53.458649+0000 2023-07-20T02:51:47.775 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/4222569264 2023-07-21T02:44:25.804258+0000 2023-07-20T02:51:47.776 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/1357065986 2023-07-21T02:44:25.804258+0000 2023-07-20T02:51:47.776 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/884368977 2023-07-21T02:44:53.458649+0000 2023-07-20T02:51:47.776 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6801/1224472946 2023-07-21T02:44:25.804258+0000 2023-07-20T02:51:47.776 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/1459293748 2023-07-21T02:44:25.804258+0000 2023-07-20T02:51:47.776 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6800/1224472946 2023-07-21T02:44:25.804258+0000 2023-07-20T02:51:47.776 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6800/2926310425 2023-07-21T02:44:53.458649+0000 2023-07-20T02:51:47.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.763+0000 7fb54cfff700 1 -- 172.21.15.7:0/466873929 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fb5300a1150 msgr2=0x7fb5300a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:47.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.763+0000 7fb54cfff700 1 --2- 172.21.15.7:0/466873929 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fb5300a1150 0x7fb5300a3610 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb53c00fa40 tx=0x7fb53c019040 comp rx=0 tx=0).stop 2023-07-20T02:51:47.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.763+0000 7fb54cfff700 1 -- 172.21.15.7:0/466873929 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb548100760 msgr2=0x7fb548074520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:47.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.763+0000 7fb54cfff700 1 --2- 172.21.15.7:0/466873929 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb548100760 0x7fb548074520 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb53800e9f0 tx=0x7fb53800ed00 comp rx=0 tx=0).stop 2023-07-20T02:51:47.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.763+0000 7fb54cfff700 1 -- 172.21.15.7:0/466873929 shutdown_connections 2023-07-20T02:51:47.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.763+0000 7fb54cfff700 1 --2- 172.21.15.7:0/466873929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb5481022f0 0x7fb5480730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:47.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.763+0000 7fb54cfff700 1 --2- 172.21.15.7:0/466873929 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fb5300a1150 0x7fb5300a3610 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:47.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.763+0000 7fb54cfff700 1 --2- 172.21.15.7:0/466873929 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb548100760 0x7fb548074520 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:47.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.763+0000 7fb54cfff700 1 --2- 172.21.15.7:0/466873929 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb548101950 0x7fb548072b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:47.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.763+0000 7fb54cfff700 1 -- 172.21.15.7:0/466873929 >> 172.21.15.7:0/466873929 conn(0x7fb5480fbcb0 msgr2=0x7fb5480fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:47.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.764+0000 7fb54cfff700 1 -- 172.21.15.7:0/466873929 shutdown_connections 2023-07-20T02:51:47.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:47.764+0000 7fb54cfff700 1 -- 172.21.15.7:0/466873929 wait complete. 2023-07-20T02:51:47.779 INFO:teuthology.orchestra.run.smithi007.stderr:listed 10 entries 2023-07-20T02:51:47.795 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-07-20T02:51:47.795 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-20T02:51:47.839 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T02:51:47.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:47 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/2975453906' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-20T02:51:47.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:47 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/1212611308' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-20T02:51:47.940 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:47 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/2975453906' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-20T02:51:47.940 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:47 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/1212611308' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-20T02:51:47.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:47 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/2975453906' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-20T02:51:47.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:47 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/1212611308' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-20T02:51:48.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.069+0000 7f6f297ff700 1 -- 172.21.15.7:0/3177293415 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f24100770 msgr2=0x7f6f24100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:48.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.069+0000 7f6f297ff700 1 --2- 172.21.15.7:0/3177293415 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f24100770 0x7f6f24100b90 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6f140099c0 tx=0x7f6f14009cd0 comp rx=0 tx=0).stop 2023-07-20T02:51:48.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.069+0000 7f6f297ff700 1 -- 172.21.15.7:0/3177293415 shutdown_connections 2023-07-20T02:51:48.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.069+0000 7f6f297ff700 1 --2- 172.21.15.7:0/3177293415 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f24102300 0x7f6f24108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:48.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.069+0000 7f6f297ff700 1 --2- 172.21.15.7:0/3177293415 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f24100770 0x7f6f24100b90 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:48.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.069+0000 7f6f297ff700 1 --2- 172.21.15.7:0/3177293415 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f24101960 0x7f6f24101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:48.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.069+0000 7f6f297ff700 1 -- 172.21.15.7:0/3177293415 >> 172.21.15.7:0/3177293415 conn(0x7f6f240fbca0 msgr2=0x7f6f240fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:48.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.069+0000 7f6f297ff700 1 -- 172.21.15.7:0/3177293415 shutdown_connections 2023-07-20T02:51:48.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.069+0000 7f6f297ff700 1 -- 172.21.15.7:0/3177293415 wait complete. 2023-07-20T02:51:48.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.070+0000 7f6f297ff700 1 Processor -- start 2023-07-20T02:51:48.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.071+0000 7f6f297ff700 1 -- start start 2023-07-20T02:51:48.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.071+0000 7f6f297ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f24100770 0x7f6f24196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:48.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.071+0000 7f6f297ff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f24101960 0x7f6f241969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:48.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.071+0000 7f6f297ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f24102300 0x7f6f2419da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:48.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.071+0000 7f6f297ff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2419dff0 con 0x7f6f24102300 2023-07-20T02:51:48.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.071+0000 7f6f297ff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2419e130 con 0x7f6f24101960 2023-07-20T02:51:48.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.071+0000 7f6f297ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2419e2c0 con 0x7f6f24100770 2023-07-20T02:51:48.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.071+0000 7f6f237fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f24102300 0x7f6f2419da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.071+0000 7f6f237fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f24102300 0x7f6f2419da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48578/0 (socket says 172.21.15.7:48578) 2023-07-20T02:51:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.071+0000 7f6f237fe700 1 -- 172.21.15.7:0/2180985436 learned_addr learned my addr 172.21.15.7:0/2180985436 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.071+0000 7f6f22ffd700 1 --2- 172.21.15.7:0/2180985436 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f24100770 0x7f6f24196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.072+0000 7f6f1bfff700 1 --2- 172.21.15.7:0/2180985436 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f24101960 0x7f6f241969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.072+0000 7f6f237fe700 1 -- 172.21.15.7:0/2180985436 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f24100770 msgr2=0x7f6f24196470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.072+0000 7f6f237fe700 1 --2- 172.21.15.7:0/2180985436 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f24100770 0x7f6f24196470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.072+0000 7f6f237fe700 1 -- 172.21.15.7:0/2180985436 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f24101960 msgr2=0x7f6f241969b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.072+0000 7f6f237fe700 1 --2- 172.21.15.7:0/2180985436 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f24101960 0x7f6f241969b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:48.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.072+0000 7f6f237fe700 1 -- 172.21.15.7:0/2180985436 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f140095f0 con 0x7f6f24102300 2023-07-20T02:51:48.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.072+0000 7f6f22ffd700 1 --2- 172.21.15.7:0/2180985436 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f24100770 0x7f6f24196470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T02:51:48.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.072+0000 7f6f237fe700 1 --2- 172.21.15.7:0/2180985436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f24102300 0x7f6f2419da20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c00ef30 tx=0x7f6f1c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:48.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.072+0000 7f6f20ff9700 1 -- 172.21.15.7:0/2180985436 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f1c018070 con 0x7f6f24102300 2023-07-20T02:51:48.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.072+0000 7f6f20ff9700 1 -- 172.21.15.7:0/2180985436 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6f1c00f040 con 0x7f6f24102300 2023-07-20T02:51:48.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.072+0000 7f6f20ff9700 1 -- 172.21.15.7:0/2180985436 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f1c00cd00 con 0x7f6f24102300 2023-07-20T02:51:48.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.072+0000 7f6f297ff700 1 -- 172.21.15.7:0/2180985436 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f2419e540 con 0x7f6f24102300 2023-07-20T02:51:48.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.073+0000 7f6f297ff700 1 -- 172.21.15.7:0/2180985436 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f2419ea00 con 0x7f6f24102300 2023-07-20T02:51:48.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.073+0000 7f6f297ff700 1 -- 172.21.15.7:0/2180985436 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f2410bfe0 con 0x7f6f24102300 2023-07-20T02:51:48.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.076+0000 7f6f20ff9700 1 -- 172.21.15.7:0/2180985436 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f6f1c003a00 con 0x7f6f24102300 2023-07-20T02:51:48.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.077+0000 7f6f20ff9700 1 --2- 172.21.15.7:0/2180985436 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f6f100a1220 0x7f6f100a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:48.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.077+0000 7f6f20ff9700 1 -- 172.21.15.7:0/2180985436 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f6f1c0cc620 con 0x7f6f24102300 2023-07-20T02:51:48.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.077+0000 7f6f22ffd700 1 --2- 172.21.15.7:0/2180985436 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f6f100a1220 0x7f6f100a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:48.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.078+0000 7f6f22ffd700 1 --2- 172.21.15.7:0/2180985436 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f6f100a1220 0x7f6f100a36e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6f14009990 tx=0x7f6f14005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:48.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.081+0000 7f6f20ff9700 1 -- 172.21.15.7:0/2180985436 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6f1c095bb0 con 0x7f6f24102300 2023-07-20T02:51:48.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.294+0000 7f6f297ff700 1 -- 172.21.15.7:0/2180985436 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f24107670 con 0x7f6f24102300 2023-07-20T02:51:48.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.296+0000 7f6f20ff9700 1 -- 172.21.15.7:0/2180985436 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+23927 (secure 0 0 0) 0x7f6f1c0991d0 con 0x7f6f24102300 2023-07-20T02:51:48.310 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:51:48.310 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":87,"fsid":"1b04643c-26a7-11ee-9b34-001a4aab830c","created":"2023-07-20T02:43:56.544719+0000","modified":"2023-07-20T02:51:33.697865+0000","last_up_change":"2023-07-20T02:50:40.126926+0000","last_in_change":"2023-07-20T02:50:21.875020+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T02:47:12.055539+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-20T02:51:26.617961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-20T02:51:28.081016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-20T02:51:31.658088+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ad6c6339-6e23-4c21-9fde-2dbecdafdb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6803","nonce":3322992091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6805","nonce":3322992091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6809","nonce":3322992091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6807","nonce":3322992091}]},"public_addr":"172.21.15.7:6803/3322992091","cluster_addr":"172.21.15.7:6805/3322992091","heartbeat_back_addr":"172.21.15.7:6809/3322992091","heartbeat_front_addr":"172.21.15.7:6807/3322992091","state":["exists","up"]},{"osd":1,"uuid":"cad207b3-a1ea-4f07-bd94-9b11fd573a69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6801","nonce":2796771612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6803","nonce":2796771612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6807","nonce":2796771612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6805","nonce":2796771612}]},"public_addr":"172.21.15.158:6801/2796771612","cluster_addr":"172.21.15.158:6803/2796771612","heartbeat_back_addr":"172.21.15.158:6807/2796771612","heartbeat_front_addr":"172.21.15.158:6805/2796771612","state":["exists","up"]},{"osd":2,"uuid":"2c777fe6-d003-49d0-af37-f63fca37ff06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6801","nonce":2928307258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6803","nonce":2928307258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6807","nonce":2928307258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6805","nonce":2928307258}]},"public_addr":"172.21.15.188:6801/2928307258","cluster_addr":"172.21.15.188:6803/2928307258","heartbeat_back_addr":"172.21.15.188:6807/2928307258","heartbeat_front_addr":"172.21.15.188:6805/2928307258","state":["exists","up"]},{"osd":3,"uuid":"59e29965-018f-4eb6-adbd-1a9e1df84245","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6811","nonce":2192953827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6813","nonce":2192953827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6817","nonce":2192953827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6815","nonce":2192953827}]},"public_addr":"172.21.15.7:6811/2192953827","cluster_addr":"172.21.15.7:6813/2192953827","heartbeat_back_addr":"172.21.15.7:6817/2192953827","heartbeat_front_addr":"172.21.15.7:6815/2192953827","state":["exists","up"]},{"osd":4,"uuid":"e2453384-3b09-4f2a-bea7-172d8fc466d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6809","nonce":3330591132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6810","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6811","nonce":3330591132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6814","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6815","nonce":3330591132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6813","nonce":3330591132}]},"public_addr":"172.21.15.158:6809/3330591132","cluster_addr":"172.21.15.158:6811/3330591132","heartbeat_back_addr":"172.21.15.158:6815/3330591132","heartbeat_front_addr":"172.21.15.158:6813/3330591132","state":["exists","up"]},{"osd":5,"uuid":"c3c04c13-0ce5-440d-9e62-0e301ad9f771","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6809","nonce":3142012805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6811","nonce":3142012805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6815","nonce":3142012805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6813","nonce":3142012805}]},"public_addr":"172.21.15.188:6809/3142012805","cluster_addr":"172.21.15.188:6811/3142012805","heartbeat_back_addr":"172.21.15.188:6815/3142012805","heartbeat_front_addr":"172.21.15.188:6813/3142012805","state":["exists","up"]},{"osd":6,"uuid":"82ef2d33-1045-41f1-bf69-b03404bcace8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6819","nonce":859560576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6821","nonce":859560576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6825","nonce":859560576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6823","nonce":859560576}]},"public_addr":"172.21.15.7:6819/859560576","cluster_addr":"172.21.15.7:6821/859560576","heartbeat_back_addr":"172.21.15.7:6825/859560576","heartbeat_front_addr":"172.21.15.7:6823/859560576","state":["exists","up"]},{"osd":7,"uuid":"020ec859-3fc2-4da2-a0dc-10530a4a366c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6817","nonce":2578758635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6819","nonce":2578758635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6822","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6823","nonce":2578758635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6821","nonce":2578758635}]},"public_addr":"172.21.15.158:6817/2578758635","cluster_addr":"172.21.15.158:6819/2578758635","heartbeat_back_addr":"172.21.15.158:6823/2578758635","heartbeat_front_addr":"172.21.15.158:6821/2578758635","state":["exists","up"]},{"osd":8,"uuid":"4081496a-45ee-4b25-8f17-a23665521187","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6817","nonce":3020824843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6819","nonce":3020824843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6823","nonce":3020824843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6821","nonce":3020824843}]},"public_addr":"172.21.15.188:6817/3020824843","cluster_addr":"172.21.15.188:6819/3020824843","heartbeat_back_addr":"172.21.15.188:6823/3020824843","heartbeat_front_addr":"172.21.15.188:6821/3020824843","state":["exists","up"]},{"osd":9,"uuid":"723580ae-efe3-4d86-95fd-d7a4c0631425","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6827","nonce":1612464825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6829","nonce":1612464825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6833","nonce":1612464825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6831","nonce":1612464825}]},"public_addr":"172.21.15.7:6827/1612464825","cluster_addr":"172.21.15.7:6829/1612464825","heartbeat_back_addr":"172.21.15.7:6833/1612464825","heartbeat_front_addr":"172.21.15.7:6831/1612464825","state":["exists","up"]},{"osd":10,"uuid":"c2651548-fe46-4d2d-b911-278036412ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6825","nonce":1653166210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6827","nonce":1653166210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6831","nonce":1653166210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6829","nonce":1653166210}]},"public_addr":"172.21.15.158:6825/1653166210","cluster_addr":"172.21.15.158:6827/1653166210","heartbeat_back_addr":"172.21.15.158:6831/1653166210","heartbeat_front_addr":"172.21.15.158:6829/1653166210","state":["exists","up"]},{"osd":11,"uuid":"f7262bcb-4243-403a-a321-fc2bdf1d4d25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6825","nonce":1230196525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6827","nonce":1230196525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6831","nonce":1230196525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6829","nonce":1230196525}]},"public_addr":"172.21.15.188:6825/1230196525","cluster_addr":"172.21.15.188:6827/1230196525","heartbeat_back_addr":"172.21.15.188:6831/1230196525","heartbeat_front_addr":"172.21.15.188:6829/1230196525","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:46:26.803832+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:46:48.726253+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:47:09.744123+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:47:36.907588+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:47:57.696685+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:48:19.120806+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:48:45.313043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:49:06.686384+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:49:28.475078+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:49:54.984033+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:50:16.541259+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:50:38.221746+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/3892248290":"2023-07-21T02:44:53.458649+0000","172.21.15.7:0/2373058722":"2023-07-21T02:44:53.458649+0000","172.21.15.7:6801/2926310425":"2023-07-21T02:44:53.458649+0000","172.21.15.7:0/4222569264":"2023-07-21T02:44:25.804258+0000","172.21.15.7:0/1357065986":"2023-07-21T02:44:25.804258+0000","172.21.15.7:0/884368977":"2023-07-21T02:44:53.458649+0000","172.21.15.7:6801/1224472946":"2023-07-21T02:44:25.804258+0000","172.21.15.7:0/1459293748":"2023-07-21T02:44:25.804258+0000","172.21.15.7:6800/1224472946":"2023-07-21T02:44:25.804258+0000","172.21.15.7:6800/2926310425":"2023-07-21T02:44:53.458649+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T02:51:48.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.298+0000 7f6f297ff700 1 -- 172.21.15.7:0/2180985436 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f6f100a1220 msgr2=0x7f6f100a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:48.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.298+0000 7f6f297ff700 1 --2- 172.21.15.7:0/2180985436 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f6f100a1220 0x7f6f100a36e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6f14009990 tx=0x7f6f14005fb0 comp rx=0 tx=0).stop 2023-07-20T02:51:48.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.298+0000 7f6f297ff700 1 -- 172.21.15.7:0/2180985436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f24102300 msgr2=0x7f6f2419da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:48.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.298+0000 7f6f297ff700 1 --2- 172.21.15.7:0/2180985436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f24102300 0x7f6f2419da20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c00ef30 tx=0x7f6f1c00c3a0 comp rx=0 tx=0).stop 2023-07-20T02:51:48.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.298+0000 7f6f297ff700 1 -- 172.21.15.7:0/2180985436 shutdown_connections 2023-07-20T02:51:48.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.298+0000 7f6f297ff700 1 --2- 172.21.15.7:0/2180985436 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f6f100a1220 0x7f6f100a36e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:48.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.298+0000 7f6f297ff700 1 --2- 172.21.15.7:0/2180985436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f24102300 0x7f6f2419da20 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:48.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.298+0000 7f6f297ff700 1 --2- 172.21.15.7:0/2180985436 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f24100770 0x7f6f24196470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:48.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.298+0000 7f6f297ff700 1 --2- 172.21.15.7:0/2180985436 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f6f24101960 0x7f6f241969b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:48.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.298+0000 7f6f297ff700 1 -- 172.21.15.7:0/2180985436 >> 172.21.15.7:0/2180985436 conn(0x7f6f240fbca0 msgr2=0x7f6f240fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:48.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.298+0000 7f6f297ff700 1 -- 172.21.15.7:0/2180985436 shutdown_connections 2023-07-20T02:51:48.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.298+0000 7f6f297ff700 1 -- 172.21.15.7:0/2180985436 wait complete. 2023-07-20T02:51:48.330 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-20T02:47:12.055539+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-20T02:51:26.617961+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-20T02:51:28.081016+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-20T02:51:31.658088+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-20T02:51:48.331 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-20T02:51:48.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.524+0000 7fc5f37ff700 1 -- 172.21.15.7:0/4155762013 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5ec1022f0 msgr2=0x7fc5ec108ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:48.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.524+0000 7fc5f37ff700 1 --2- 172.21.15.7:0/4155762013 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5ec1022f0 0x7fc5ec108ba0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc5e400b0d0 tx=0x7fc5e400b3e0 comp rx=0 tx=0).stop 2023-07-20T02:51:48.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.524+0000 7fc5f37ff700 1 -- 172.21.15.7:0/4155762013 shutdown_connections 2023-07-20T02:51:48.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.524+0000 7fc5f37ff700 1 --2- 172.21.15.7:0/4155762013 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5ec1022f0 0x7fc5ec108ba0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:48.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.524+0000 7fc5f37ff700 1 --2- 172.21.15.7:0/4155762013 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5ec100760 0x7fc5ec100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:48.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.524+0000 7fc5f37ff700 1 --2- 172.21.15.7:0/4155762013 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5ec101950 0x7fc5ec101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:48.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.524+0000 7fc5f37ff700 1 -- 172.21.15.7:0/4155762013 >> 172.21.15.7:0/4155762013 conn(0x7fc5ec0fbcb0 msgr2=0x7fc5ec0fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:48.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.524+0000 7fc5f37ff700 1 -- 172.21.15.7:0/4155762013 shutdown_connections 2023-07-20T02:51:48.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.525+0000 7fc5f37ff700 1 -- 172.21.15.7:0/4155762013 wait complete. 2023-07-20T02:51:48.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.526+0000 7fc5f37ff700 1 Processor -- start 2023-07-20T02:51:48.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.526+0000 7fc5f37ff700 1 -- start start 2023-07-20T02:51:48.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.527+0000 7fc5f37ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5ec100760 0x7fc5ec074520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:48.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.527+0000 7fc5f37ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5ec101950 0x7fc5ec072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:48.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.527+0000 7fc5f37ff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5ec1022f0 0x7fc5ec0730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:48.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.527+0000 7fc5f37ff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec073840 con 0x7fc5ec100760 2023-07-20T02:51:48.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.527+0000 7fc5f37ff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec0739b0 con 0x7fc5ec1022f0 2023-07-20T02:51:48.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.527+0000 7fc5f37ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec1a05b0 con 0x7fc5ec101950 2023-07-20T02:51:48.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.527+0000 7fc5f159b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5ec100760 0x7fc5ec074520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:48.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.527+0000 7fc5f159b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5ec100760 0x7fc5ec074520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48596/0 (socket says 172.21.15.7:48596) 2023-07-20T02:51:48.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.527+0000 7fc5f159b700 1 -- 172.21.15.7:0/1667142183 learned_addr learned my addr 172.21.15.7:0/1667142183 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:48.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.527+0000 7fc5f0d9a700 1 --2- 172.21.15.7:0/1667142183 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5ec101950 0x7fc5ec072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:48.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.527+0000 7fc5f1d9c700 1 --2- 172.21.15.7:0/1667142183 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5ec1022f0 0x7fc5ec0730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:48.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.527+0000 7fc5f159b700 1 -- 172.21.15.7:0/1667142183 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5ec101950 msgr2=0x7fc5ec072b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:48.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.527+0000 7fc5f159b700 1 --2- 172.21.15.7:0/1667142183 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5ec101950 0x7fc5ec072b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:48.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.527+0000 7fc5f159b700 1 -- 172.21.15.7:0/1667142183 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5ec1022f0 msgr2=0x7fc5ec0730a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:48.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.527+0000 7fc5f159b700 1 --2- 172.21.15.7:0/1667142183 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5ec1022f0 0x7fc5ec0730a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:48.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.527+0000 7fc5f159b700 1 -- 172.21.15.7:0/1667142183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e4009d00 con 0x7fc5ec100760 2023-07-20T02:51:48.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.528+0000 7fc5f159b700 1 --2- 172.21.15.7:0/1667142183 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5ec100760 0x7fc5ec074520 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc5e800e970 tx=0x7fc5e800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:48.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.528+0000 7fc5e27fc700 1 -- 172.21.15.7:0/1667142183 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e8009790 con 0x7fc5ec100760 2023-07-20T02:51:48.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.528+0000 7fc5f37ff700 1 -- 172.21.15.7:0/1667142183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5ec1a0890 con 0x7fc5ec100760 2023-07-20T02:51:48.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.528+0000 7fc5e27fc700 1 -- 172.21.15.7:0/1667142183 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc5e800c7e0 con 0x7fc5ec100760 2023-07-20T02:51:48.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.528+0000 7fc5e27fc700 1 -- 172.21.15.7:0/1667142183 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e8003c70 con 0x7fc5ec100760 2023-07-20T02:51:48.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.528+0000 7fc5f37ff700 1 -- 172.21.15.7:0/1667142183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5ec1a0df0 con 0x7fc5ec100760 2023-07-20T02:51:48.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.529+0000 7fc5f37ff700 1 -- 172.21.15.7:0/1667142183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5ec107ca0 con 0x7fc5ec100760 2023-07-20T02:51:48.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.531+0000 7fc5e27fc700 1 -- 172.21.15.7:0/1667142183 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fc5e800cc50 con 0x7fc5ec100760 2023-07-20T02:51:48.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.532+0000 7fc5e27fc700 1 --2- 172.21.15.7:0/1667142183 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fc5d80a11d0 0x7fc5d80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:48.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.533+0000 7fc5e27fc700 1 -- 172.21.15.7:0/1667142183 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fc5e8014070 con 0x7fc5ec100760 2023-07-20T02:51:48.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.533+0000 7fc5f0d9a700 1 --2- 172.21.15.7:0/1667142183 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fc5d80a11d0 0x7fc5d80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:48.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.533+0000 7fc5f0d9a700 1 --2- 172.21.15.7:0/1667142183 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fc5d80a11d0 0x7fc5d80a3690 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc009c60 tx=0x7fc5dc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:48.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.536+0000 7fc5e27fc700 1 -- 172.21.15.7:0/1667142183 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc5e80961a0 con 0x7fc5ec100760 2023-07-20T02:51:48.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.747+0000 7fc5f37ff700 1 -- 172.21.15.7:0/1667142183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc5ec061420 con 0x7fc5ec100760 2023-07-20T02:51:48.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.748+0000 7fc5e27fc700 1 -- 172.21.15.7:0/1667142183 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc5e8003dd0 con 0x7fc5ec100760 2023-07-20T02:51:48.762 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2023-07-20T02:51:48.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.750+0000 7fc5f37ff700 1 -- 172.21.15.7:0/1667142183 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fc5d80a11d0 msgr2=0x7fc5d80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:48.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.750+0000 7fc5f37ff700 1 --2- 172.21.15.7:0/1667142183 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fc5d80a11d0 0x7fc5d80a3690 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc009c60 tx=0x7fc5dc009260 comp rx=0 tx=0).stop 2023-07-20T02:51:48.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.750+0000 7fc5f37ff700 1 -- 172.21.15.7:0/1667142183 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5ec100760 msgr2=0x7fc5ec074520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:48.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.750+0000 7fc5f37ff700 1 --2- 172.21.15.7:0/1667142183 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5ec100760 0x7fc5ec074520 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc5e800e970 tx=0x7fc5e800ec80 comp rx=0 tx=0).stop 2023-07-20T02:51:48.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.750+0000 7fc5f37ff700 1 -- 172.21.15.7:0/1667142183 shutdown_connections 2023-07-20T02:51:48.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.750+0000 7fc5f37ff700 1 --2- 172.21.15.7:0/1667142183 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc5ec1022f0 0x7fc5ec0730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:48.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.750+0000 7fc5f37ff700 1 --2- 172.21.15.7:0/1667142183 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fc5d80a11d0 0x7fc5d80a3690 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:48.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.750+0000 7fc5f37ff700 1 --2- 172.21.15.7:0/1667142183 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5ec100760 0x7fc5ec074520 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:48.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.750+0000 7fc5f37ff700 1 --2- 172.21.15.7:0/1667142183 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc5ec101950 0x7fc5ec072b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:48.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.750+0000 7fc5f37ff700 1 -- 172.21.15.7:0/1667142183 >> 172.21.15.7:0/1667142183 conn(0x7fc5ec0fbcb0 msgr2=0x7fc5ec104de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:48.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.751+0000 7fc5f37ff700 1 -- 172.21.15.7:0/1667142183 shutdown_connections 2023-07-20T02:51:48.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.751+0000 7fc5f37ff700 1 -- 172.21.15.7:0/1667142183 wait complete. 2023-07-20T02:51:48.783 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-20T02:51:48.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:48 smithi158 ceph-mon[128925]: pgmap v284: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-20T02:51:48.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:48 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/466873929' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-20T02:51:48.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:48 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/2180985436' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:51:48.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:48 smithi007 ceph-mon[126353]: pgmap v284: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-20T02:51:48.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:48 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/466873929' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-20T02:51:48.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:48 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/2180985436' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:51:48.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:48 smithi188 ceph-mon[129553]: pgmap v284: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-20T02:51:48.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:48 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/466873929' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-20T02:51:48.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:48 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/2180985436' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:51:48.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.976+0000 7fb145bff700 1 -- 172.21.15.7:0/2204774507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb140101950 msgr2=0x7fb140101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:48.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.976+0000 7fb145bff700 1 --2- 172.21.15.7:0/2204774507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb140101950 0x7fb140101db0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb1340099c0 tx=0x7fb134009cd0 comp rx=0 tx=0).stop 2023-07-20T02:51:48.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.976+0000 7fb145bff700 1 -- 172.21.15.7:0/2204774507 shutdown_connections 2023-07-20T02:51:48.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.976+0000 7fb145bff700 1 --2- 172.21.15.7:0/2204774507 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1401022f0 0x7fb140108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:48.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.976+0000 7fb145bff700 1 --2- 172.21.15.7:0/2204774507 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb140100760 0x7fb140100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:48.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.976+0000 7fb145bff700 1 --2- 172.21.15.7:0/2204774507 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb140101950 0x7fb140101db0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:48.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.976+0000 7fb145bff700 1 -- 172.21.15.7:0/2204774507 >> 172.21.15.7:0/2204774507 conn(0x7fb1400fbcb0 msgr2=0x7fb1400fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:48.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.977+0000 7fb145bff700 1 -- 172.21.15.7:0/2204774507 shutdown_connections 2023-07-20T02:51:48.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.977+0000 7fb145bff700 1 -- 172.21.15.7:0/2204774507 wait complete. 2023-07-20T02:51:48.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.977+0000 7fb145bff700 1 Processor -- start 2023-07-20T02:51:48.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.978+0000 7fb145bff700 1 -- start start 2023-07-20T02:51:48.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.978+0000 7fb145bff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb140100760 0x7fb140196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:48.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.978+0000 7fb145bff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb140101950 0x7fb1401969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:48.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.978+0000 7fb145bff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1401022f0 0x7fb14019da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:48.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.978+0000 7fb145bff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14019dff0 con 0x7fb1401022f0 2023-07-20T02:51:48.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.978+0000 7fb145bff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14019e130 con 0x7fb140101950 2023-07-20T02:51:48.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.978+0000 7fb145bff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14019e2c0 con 0x7fb140100760 2023-07-20T02:51:48.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.978+0000 7fb13ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1401022f0 0x7fb14019da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:48.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.978+0000 7fb13f7fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb140100760 0x7fb140196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:48.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.978+0000 7fb13f7fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb140100760 0x7fb140196470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.7:43786/0 (socket says 172.21.15.7:43786) 2023-07-20T02:51:48.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.978+0000 7fb13f7fe700 1 -- 172.21.15.7:0/3043997945 learned_addr learned my addr 172.21.15.7:0/3043997945 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:49.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.978+0000 7fb13effd700 1 --2- 172.21.15.7:0/3043997945 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb140101950 0x7fb1401969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:49.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.979+0000 7fb13f7fe700 1 -- 172.21.15.7:0/3043997945 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb140101950 msgr2=0x7fb1401969b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:49.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.979+0000 7fb13f7fe700 1 --2- 172.21.15.7:0/3043997945 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb140101950 0x7fb1401969b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:49.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.979+0000 7fb13f7fe700 1 -- 172.21.15.7:0/3043997945 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1401022f0 msgr2=0x7fb14019da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:49.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.979+0000 7fb13f7fe700 1 --2- 172.21.15.7:0/3043997945 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1401022f0 0x7fb14019da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:49.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.979+0000 7fb13f7fe700 1 -- 172.21.15.7:0/3043997945 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1340095f0 con 0x7fb140100760 2023-07-20T02:51:49.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.979+0000 7fb13f7fe700 1 --2- 172.21.15.7:0/3043997945 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb140100760 0x7fb140196470 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb13000e970 tx=0x7fb13000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:49.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.979+0000 7fb13cff9700 1 -- 172.21.15.7:0/3043997945 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb130009790 con 0x7fb140100760 2023-07-20T02:51:49.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.979+0000 7fb145bff700 1 -- 172.21.15.7:0/3043997945 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb14019e5a0 con 0x7fb140100760 2023-07-20T02:51:49.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.979+0000 7fb13cff9700 1 -- 172.21.15.7:0/3043997945 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb13000eec0 con 0x7fb140100760 2023-07-20T02:51:49.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.979+0000 7fb13cff9700 1 -- 172.21.15.7:0/3043997945 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb130003cf0 con 0x7fb140100760 2023-07-20T02:51:49.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.979+0000 7fb145bff700 1 -- 172.21.15.7:0/3043997945 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb14019eb00 con 0x7fb140100760 2023-07-20T02:51:49.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.980+0000 7fb145bff700 1 -- 172.21.15.7:0/3043997945 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1401076e0 con 0x7fb140100760 2023-07-20T02:51:49.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.982+0000 7fb13cff9700 1 -- 172.21.15.7:0/3043997945 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fb13000ccd0 con 0x7fb140100760 2023-07-20T02:51:49.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.984+0000 7fb13cff9700 1 --2- 172.21.15.7:0/3043997945 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fb1280a1150 0x7fb1280a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:49.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.984+0000 7fb13cff9700 1 -- 172.21.15.7:0/3043997945 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fb130014070 con 0x7fb140100760 2023-07-20T02:51:49.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.984+0000 7fb13effd700 1 --2- 172.21.15.7:0/3043997945 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fb1280a1150 0x7fb1280a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:49.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.985+0000 7fb13effd700 1 --2- 172.21.15.7:0/3043997945 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fb1280a1150 0x7fb1280a3610 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb13400b580 tx=0x7fb13401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:49.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:48.986+0000 7fb13cff9700 1 -- 172.21.15.7:0/3043997945 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb1300963f0 con 0x7fb140100760 2023-07-20T02:51:49.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.195+0000 7fb145bff700 1 -- 172.21.15.7:0/3043997945 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb14002cc00 con 0x7fb140100760 2023-07-20T02:51:49.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.195+0000 7fb13cff9700 1 -- 172.21.15.7:0/3043997945 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb130099a10 con 0x7fb140100760 2023-07-20T02:51:49.209 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2023-07-20T02:51:49.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.198+0000 7fb145bff700 1 -- 172.21.15.7:0/3043997945 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fb1280a1150 msgr2=0x7fb1280a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:49.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.198+0000 7fb145bff700 1 --2- 172.21.15.7:0/3043997945 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fb1280a1150 0x7fb1280a3610 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb13400b580 tx=0x7fb13401a040 comp rx=0 tx=0).stop 2023-07-20T02:51:49.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.198+0000 7fb145bff700 1 -- 172.21.15.7:0/3043997945 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb140100760 msgr2=0x7fb140196470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:49.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.198+0000 7fb145bff700 1 --2- 172.21.15.7:0/3043997945 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb140100760 0x7fb140196470 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb13000e970 tx=0x7fb13000ec80 comp rx=0 tx=0).stop 2023-07-20T02:51:49.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.199+0000 7fb145bff700 1 -- 172.21.15.7:0/3043997945 shutdown_connections 2023-07-20T02:51:49.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.199+0000 7fb145bff700 1 --2- 172.21.15.7:0/3043997945 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fb1280a1150 0x7fb1280a3610 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:49.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.199+0000 7fb145bff700 1 --2- 172.21.15.7:0/3043997945 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1401022f0 0x7fb14019da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:49.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.199+0000 7fb145bff700 1 --2- 172.21.15.7:0/3043997945 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb140100760 0x7fb140196470 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:49.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.199+0000 7fb145bff700 1 --2- 172.21.15.7:0/3043997945 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb140101950 0x7fb1401969b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:49.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.199+0000 7fb145bff700 1 -- 172.21.15.7:0/3043997945 >> 172.21.15.7:0/3043997945 conn(0x7fb1400fbcb0 msgr2=0x7fb1400fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:49.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.199+0000 7fb145bff700 1 -- 172.21.15.7:0/3043997945 shutdown_connections 2023-07-20T02:51:49.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.199+0000 7fb145bff700 1 -- 172.21.15.7:0/3043997945 wait complete. 2023-07-20T02:51:49.231 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-20T02:51:49.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.423+0000 7f26d4fff700 1 -- 172.21.15.7:0/3202138456 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26d0074750 msgr2=0x7f26d0072ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:49.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.423+0000 7f26d4fff700 1 --2- 172.21.15.7:0/3202138456 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26d0074750 0x7f26d0072ba0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f26c0009990 tx=0x7f26c0009ca0 comp rx=0 tx=0).stop 2023-07-20T02:51:49.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.423+0000 7f26d4fff700 1 -- 172.21.15.7:0/3202138456 shutdown_connections 2023-07-20T02:51:49.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.423+0000 7f26d4fff700 1 --2- 172.21.15.7:0/3202138456 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f26d0108630 0x7f26d010aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:49.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.423+0000 7f26d4fff700 1 --2- 172.21.15.7:0/3202138456 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26d0074750 0x7f26d0072ba0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:49.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.423+0000 7f26d4fff700 1 --2- 172.21.15.7:0/3202138456 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26d0073170 0x7f26d00735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:49.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.423+0000 7f26d4fff700 1 -- 172.21.15.7:0/3202138456 >> 172.21.15.7:0/3202138456 conn(0x7f26d00fbcb0 msgr2=0x7f26d00fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:49.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.423+0000 7f26d4fff700 1 -- 172.21.15.7:0/3202138456 shutdown_connections 2023-07-20T02:51:49.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.423+0000 7f26d4fff700 1 -- 172.21.15.7:0/3202138456 wait complete. 2023-07-20T02:51:49.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.424+0000 7f26d4fff700 1 Processor -- start 2023-07-20T02:51:49.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.424+0000 7f26d4fff700 1 -- start start 2023-07-20T02:51:49.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.425+0000 7f26d4fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26d0073170 0x7f26d0198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:49.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.425+0000 7f26d4fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f26d0074750 0x7f26d0198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:49.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.425+0000 7f26d4fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26d0108630 0x7f26d019fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:49.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.425+0000 7f26d4fff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d01a0210 con 0x7f26d0108630 2023-07-20T02:51:49.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.425+0000 7f26d4fff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d01a0350 con 0x7f26d0073170 2023-07-20T02:51:49.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.425+0000 7f26d4fff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d01a04e0 con 0x7f26d0074750 2023-07-20T02:51:49.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.425+0000 7f26ced9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26d0108630 0x7f26d019fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:49.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.425+0000 7f26ced9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26d0108630 0x7f26d019fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48644/0 (socket says 172.21.15.7:48644) 2023-07-20T02:51:49.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.425+0000 7f26ced9d700 1 -- 172.21.15.7:0/3204411248 learned_addr learned my addr 172.21.15.7:0/3204411248 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:49.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.425+0000 7f26cdd9b700 1 --2- 172.21.15.7:0/3204411248 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f26d0074750 0x7f26d0198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:49.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.425+0000 7f26ce59c700 1 --2- 172.21.15.7:0/3204411248 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26d0073170 0x7f26d0198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:49.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.425+0000 7f26ced9d700 1 -- 172.21.15.7:0/3204411248 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f26d0074750 msgr2=0x7f26d0198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:49.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.425+0000 7f26ced9d700 1 --2- 172.21.15.7:0/3204411248 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f26d0074750 0x7f26d0198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:49.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.425+0000 7f26ced9d700 1 -- 172.21.15.7:0/3204411248 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26d0073170 msgr2=0x7f26d0198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:49.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.425+0000 7f26ced9d700 1 --2- 172.21.15.7:0/3204411248 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26d0073170 0x7f26d0198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:49.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.425+0000 7f26ced9d700 1 -- 172.21.15.7:0/3204411248 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c00095f0 con 0x7f26d0108630 2023-07-20T02:51:49.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.426+0000 7f26ced9d700 1 --2- 172.21.15.7:0/3204411248 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26d0108630 0x7f26d019fc40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f26c800ef30 tx=0x7f26c800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:49.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.426+0000 7f26bf7fe700 1 -- 172.21.15.7:0/3204411248 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c8018070 con 0x7f26d0108630 2023-07-20T02:51:49.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.426+0000 7f26bf7fe700 1 -- 172.21.15.7:0/3204411248 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f26c800f040 con 0x7f26d0108630 2023-07-20T02:51:49.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.426+0000 7f26d4fff700 1 -- 172.21.15.7:0/3204411248 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26d01a0760 con 0x7f26d0108630 2023-07-20T02:51:49.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.426+0000 7f26bf7fe700 1 -- 172.21.15.7:0/3204411248 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c800cd00 con 0x7f26d0108630 2023-07-20T02:51:49.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.426+0000 7f26d4fff700 1 -- 172.21.15.7:0/3204411248 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26d01a0c70 con 0x7f26d0108630 2023-07-20T02:51:49.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.427+0000 7f26d4fff700 1 -- 172.21.15.7:0/3204411248 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26d0106d30 con 0x7f26d0108630 2023-07-20T02:51:49.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.429+0000 7f26bf7fe700 1 -- 172.21.15.7:0/3204411248 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f26c8003a00 con 0x7f26d0108630 2023-07-20T02:51:49.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.430+0000 7f26bf7fe700 1 --2- 172.21.15.7:0/3204411248 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f26b80a11d0 0x7f26b80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:49.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.430+0000 7f26bf7fe700 1 -- 172.21.15.7:0/3204411248 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f26c80cc620 con 0x7f26d0108630 2023-07-20T02:51:49.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.431+0000 7f26ce59c700 1 --2- 172.21.15.7:0/3204411248 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f26b80a11d0 0x7f26b80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:49.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.431+0000 7f26ce59c700 1 --2- 172.21.15.7:0/3204411248 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f26b80a11d0 0x7f26b80a3690 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f26c0006010 tx=0x7f26c0005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:49.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.434+0000 7f26bf7fe700 1 -- 172.21.15.7:0/3204411248 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f26c8095bb0 con 0x7f26d0108630 2023-07-20T02:51:49.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.643+0000 7f26d4fff700 1 -- 172.21.15.7:0/3204411248 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f26d0061380 con 0x7f26d0108630 2023-07-20T02:51:49.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.643+0000 7f26bf7fe700 1 -- 172.21.15.7:0/3204411248 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f26c80991d0 con 0x7f26d0108630 2023-07-20T02:51:49.657 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2023-07-20T02:51:49.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.645+0000 7f26d4fff700 1 -- 172.21.15.7:0/3204411248 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f26b80a11d0 msgr2=0x7f26b80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:49.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.645+0000 7f26d4fff700 1 --2- 172.21.15.7:0/3204411248 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f26b80a11d0 0x7f26b80a3690 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f26c0006010 tx=0x7f26c0005a90 comp rx=0 tx=0).stop 2023-07-20T02:51:49.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.645+0000 7f26d4fff700 1 -- 172.21.15.7:0/3204411248 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26d0108630 msgr2=0x7f26d019fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:49.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.645+0000 7f26d4fff700 1 --2- 172.21.15.7:0/3204411248 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26d0108630 0x7f26d019fc40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f26c800ef30 tx=0x7f26c800c3a0 comp rx=0 tx=0).stop 2023-07-20T02:51:49.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.645+0000 7f26d4fff700 1 -- 172.21.15.7:0/3204411248 shutdown_connections 2023-07-20T02:51:49.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.645+0000 7f26d4fff700 1 --2- 172.21.15.7:0/3204411248 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f26b80a11d0 0x7f26b80a3690 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:49.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.645+0000 7f26d4fff700 1 --2- 172.21.15.7:0/3204411248 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26d0108630 0x7f26d019fc40 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:49.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.645+0000 7f26d4fff700 1 --2- 172.21.15.7:0/3204411248 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f26d0073170 0x7f26d0198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:49.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.645+0000 7f26d4fff700 1 --2- 172.21.15.7:0/3204411248 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f26d0074750 0x7f26d0198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:49.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.645+0000 7f26d4fff700 1 -- 172.21.15.7:0/3204411248 >> 172.21.15.7:0/3204411248 conn(0x7f26d00fbcb0 msgr2=0x7f26d00fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:49.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.646+0000 7f26d4fff700 1 -- 172.21.15.7:0/3204411248 shutdown_connections 2023-07-20T02:51:49.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.646+0000 7f26d4fff700 1 -- 172.21.15.7:0/3204411248 wait complete. 2023-07-20T02:51:49.678 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-20T02:51:49.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.870+0000 7f26497ff700 1 -- 172.21.15.7:0/2350145143 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2644074750 msgr2=0x7f2644072ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:49.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.870+0000 7f26497ff700 1 --2- 172.21.15.7:0/2350145143 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2644074750 0x7f2644072ba0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2634009990 tx=0x7f2634009ca0 comp rx=0 tx=0).stop 2023-07-20T02:51:49.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.870+0000 7f26497ff700 1 -- 172.21.15.7:0/2350145143 shutdown_connections 2023-07-20T02:51:49.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.870+0000 7f26497ff700 1 --2- 172.21.15.7:0/2350145143 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2644108630 0x7f264410aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:49.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.870+0000 7f26497ff700 1 --2- 172.21.15.7:0/2350145143 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2644074750 0x7f2644072ba0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:49.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.870+0000 7f26497ff700 1 --2- 172.21.15.7:0/2350145143 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2644073170 0x7f26440735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:49.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.870+0000 7f26497ff700 1 -- 172.21.15.7:0/2350145143 >> 172.21.15.7:0/2350145143 conn(0x7f26440fbcb0 msgr2=0x7f26440fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:49.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.870+0000 7f26497ff700 1 -- 172.21.15.7:0/2350145143 shutdown_connections 2023-07-20T02:51:49.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.870+0000 7f26497ff700 1 -- 172.21.15.7:0/2350145143 wait complete. 2023-07-20T02:51:49.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.871+0000 7f26497ff700 1 Processor -- start 2023-07-20T02:51:49.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.871+0000 7f26497ff700 1 -- start start 2023-07-20T02:51:49.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.872+0000 7f26497ff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2644073170 0x7f2644198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:49.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.872+0000 7f26497ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2644074750 0x7f2644198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:49.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.872+0000 7f26497ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2644108630 0x7f264419fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:49.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.872+0000 7f26497ff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26441a0200 con 0x7f2644074750 2023-07-20T02:51:49.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.872+0000 7f26497ff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26441a0340 con 0x7f2644073170 2023-07-20T02:51:49.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.872+0000 7f26497ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26441a04d0 con 0x7f2644108630 2023-07-20T02:51:49.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.872+0000 7f26427fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2644074750 0x7f2644198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:49.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.872+0000 7f26427fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2644074750 0x7f2644198bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48658/0 (socket says 172.21.15.7:48658) 2023-07-20T02:51:49.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.872+0000 7f2642ffd700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2644073170 0x7f2644198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:49.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.872+0000 7f26437fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2644108630 0x7f264419fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:49.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.872+0000 7f26427fc700 1 -- 172.21.15.7:0/1010169436 learned_addr learned my addr 172.21.15.7:0/1010169436 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:49.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.872+0000 7f26427fc700 1 -- 172.21.15.7:0/1010169436 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2644108630 msgr2=0x7f264419fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:49.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.872+0000 7f26427fc700 1 --2- 172.21.15.7:0/1010169436 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2644108630 0x7f264419fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:49.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.872+0000 7f26427fc700 1 -- 172.21.15.7:0/1010169436 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2644073170 msgr2=0x7f2644198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:49.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.872+0000 7f26427fc700 1 --2- 172.21.15.7:0/1010169436 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2644073170 0x7f2644198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:49.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.872+0000 7f26427fc700 1 -- 172.21.15.7:0/1010169436 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26340095f0 con 0x7f2644074750 2023-07-20T02:51:49.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.873+0000 7f26427fc700 1 --2- 172.21.15.7:0/1010169436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2644074750 0x7f2644198bc0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f263800e9a0 tx=0x7f263800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:49.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.873+0000 7f262bfff700 1 -- 172.21.15.7:0/1010169436 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2638009790 con 0x7f2644074750 2023-07-20T02:51:49.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.873+0000 7f26497ff700 1 -- 172.21.15.7:0/1010169436 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26441a07b0 con 0x7f2644074750 2023-07-20T02:51:49.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.873+0000 7f262bfff700 1 -- 172.21.15.7:0/1010169436 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2638010430 con 0x7f2644074750 2023-07-20T02:51:49.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.873+0000 7f262bfff700 1 -- 172.21.15.7:0/1010169436 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2638018680 con 0x7f2644074750 2023-07-20T02:51:49.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.873+0000 7f26497ff700 1 -- 172.21.15.7:0/1010169436 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26441a0d10 con 0x7f2644074750 2023-07-20T02:51:49.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.874+0000 7f26497ff700 1 -- 172.21.15.7:0/1010169436 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2644106d30 con 0x7f2644074750 2023-07-20T02:51:49.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:49 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/1667142183' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T02:51:49.892 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:49 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/3043997945' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-20T02:51:49.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.876+0000 7f262bfff700 1 -- 172.21.15.7:0/1010169436 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f26380105a0 con 0x7f2644074750 2023-07-20T02:51:49.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.877+0000 7f262bfff700 1 --2- 172.21.15.7:0/1010169436 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f262c0a1150 0x7f262c0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:49.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.877+0000 7f262bfff700 1 -- 172.21.15.7:0/1010169436 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f2638014070 con 0x7f2644074750 2023-07-20T02:51:49.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.877+0000 7f2642ffd700 1 --2- 172.21.15.7:0/1010169436 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f262c0a1150 0x7f262c0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:49.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.878+0000 7f2642ffd700 1 --2- 172.21.15.7:0/1010169436 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f262c0a1150 0x7f262c0a3610 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f263400b2e0 tx=0x7f2634005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:49.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:49.881+0000 7f262bfff700 1 -- 172.21.15.7:0/1010169436 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2638051780 con 0x7f2644074750 2023-07-20T02:51:49.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:49 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/1667142183' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T02:51:49.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:49 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/3043997945' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-20T02:51:49.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:49 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/1667142183' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T02:51:49.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:49 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/3043997945' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-20T02:51:50.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.090+0000 7f26497ff700 1 -- 172.21.15.7:0/1010169436 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f2644061380 con 0x7f2644074750 2023-07-20T02:51:50.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.091+0000 7f262bfff700 1 -- 172.21.15.7:0/1010169436 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v87) v1 ==== 103+0+11 (secure 0 0 0) 0x7f2638096220 con 0x7f2644074750 2023-07-20T02:51:50.105 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2023-07-20T02:51:50.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.093+0000 7f26497ff700 1 -- 172.21.15.7:0/1010169436 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f262c0a1150 msgr2=0x7f262c0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:50.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.093+0000 7f26497ff700 1 --2- 172.21.15.7:0/1010169436 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f262c0a1150 0x7f262c0a3610 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f263400b2e0 tx=0x7f2634005fb0 comp rx=0 tx=0).stop 2023-07-20T02:51:50.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.093+0000 7f26497ff700 1 -- 172.21.15.7:0/1010169436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2644074750 msgr2=0x7f2644198bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:50.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.093+0000 7f26497ff700 1 --2- 172.21.15.7:0/1010169436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2644074750 0x7f2644198bc0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f263800e9a0 tx=0x7f263800ecb0 comp rx=0 tx=0).stop 2023-07-20T02:51:50.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.093+0000 7f26497ff700 1 -- 172.21.15.7:0/1010169436 shutdown_connections 2023-07-20T02:51:50.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.093+0000 7f26497ff700 1 --2- 172.21.15.7:0/1010169436 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2644108630 0x7f264419fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:50.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.093+0000 7f26497ff700 1 --2- 172.21.15.7:0/1010169436 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f2644073170 0x7f2644198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:50.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.093+0000 7f26497ff700 1 --2- 172.21.15.7:0/1010169436 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f262c0a1150 0x7f262c0a3610 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:50.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.093+0000 7f26497ff700 1 --2- 172.21.15.7:0/1010169436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2644074750 0x7f2644198bc0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:50.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.093+0000 7f26497ff700 1 -- 172.21.15.7:0/1010169436 >> 172.21.15.7:0/1010169436 conn(0x7f26440fbcb0 msgr2=0x7f26440fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:50.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.093+0000 7f26497ff700 1 -- 172.21.15.7:0/1010169436 shutdown_connections 2023-07-20T02:51:50.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.093+0000 7f26497ff700 1 -- 172.21.15.7:0/1010169436 wait complete. 2023-07-20T02:51:50.126 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-07-20T02:51:50.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.318+0000 7f445a3ff700 1 -- 172.21.15.7:0/737971590 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4454101950 msgr2=0x7f4454101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:50.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.318+0000 7f445a3ff700 1 --2- 172.21.15.7:0/737971590 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4454101950 0x7f4454101db0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4448009920 tx=0x7f4448009c30 comp rx=0 tx=0).stop 2023-07-20T02:51:50.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.318+0000 7f445a3ff700 1 -- 172.21.15.7:0/737971590 shutdown_connections 2023-07-20T02:51:50.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.318+0000 7f445a3ff700 1 --2- 172.21.15.7:0/737971590 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44541022f0 0x7f4454108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:50.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.318+0000 7f445a3ff700 1 --2- 172.21.15.7:0/737971590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4454100760 0x7f4454100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:50.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.318+0000 7f445a3ff700 1 --2- 172.21.15.7:0/737971590 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4454101950 0x7f4454101db0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:50.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.318+0000 7f445a3ff700 1 -- 172.21.15.7:0/737971590 >> 172.21.15.7:0/737971590 conn(0x7f44540fbcb0 msgr2=0x7f44540fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:50.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.318+0000 7f445a3ff700 1 -- 172.21.15.7:0/737971590 shutdown_connections 2023-07-20T02:51:50.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.318+0000 7f445a3ff700 1 -- 172.21.15.7:0/737971590 wait complete. 2023-07-20T02:51:50.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.319+0000 7f445a3ff700 1 Processor -- start 2023-07-20T02:51:50.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.319+0000 7f445a3ff700 1 -- start start 2023-07-20T02:51:50.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.319+0000 7f445a3ff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4454100760 0x7f4454198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:50.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.319+0000 7f445a3ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4454101950 0x7f4454198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:50.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.319+0000 7f445a3ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44541022f0 0x7f445419fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:50.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.319+0000 7f445a3ff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44541a0280 con 0x7f4454101950 2023-07-20T02:51:50.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.320+0000 7f445a3ff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44541a03c0 con 0x7f4454100760 2023-07-20T02:51:50.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.320+0000 7f445a3ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44541a0550 con 0x7f44541022f0 2023-07-20T02:51:50.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.320+0000 7f44537fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4454101950 0x7f4454198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:50.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.320+0000 7f4453fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4454100760 0x7f4454198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:50.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.320+0000 7f44537fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4454101950 0x7f4454198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48676/0 (socket says 172.21.15.7:48676) 2023-07-20T02:51:50.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.320+0000 7f44537fe700 1 -- 172.21.15.7:0/3171141306 learned_addr learned my addr 172.21.15.7:0/3171141306 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:50.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.320+0000 7f445899c700 1 --2- 172.21.15.7:0/3171141306 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44541022f0 0x7f445419fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:50.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.320+0000 7f44537fe700 1 -- 172.21.15.7:0/3171141306 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44541022f0 msgr2=0x7f445419fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:50.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.320+0000 7f44537fe700 1 --2- 172.21.15.7:0/3171141306 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44541022f0 0x7f445419fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:50.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.320+0000 7f44537fe700 1 -- 172.21.15.7:0/3171141306 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4454100760 msgr2=0x7f4454198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:50.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.320+0000 7f44537fe700 1 --2- 172.21.15.7:0/3171141306 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4454100760 0x7f4454198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:50.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.320+0000 7f44537fe700 1 -- 172.21.15.7:0/3171141306 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44440095f0 con 0x7f4454101950 2023-07-20T02:51:50.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.320+0000 7f44537fe700 1 --2- 172.21.15.7:0/3171141306 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4454101950 0x7f4454198c40 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4448009230 tx=0x7f444800bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:50.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.321+0000 7f44517fa700 1 -- 172.21.15.7:0/3171141306 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444801c070 con 0x7f4454101950 2023-07-20T02:51:50.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.321+0000 7f445a3ff700 1 -- 172.21.15.7:0/3171141306 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4448009520 con 0x7f4454101950 2023-07-20T02:51:50.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.321+0000 7f44517fa700 1 -- 172.21.15.7:0/3171141306 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4448005040 con 0x7f4454101950 2023-07-20T02:51:50.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.321+0000 7f44517fa700 1 -- 172.21.15.7:0/3171141306 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4448003aa0 con 0x7f4454101950 2023-07-20T02:51:50.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.321+0000 7f445a3ff700 1 -- 172.21.15.7:0/3171141306 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44541a0ac0 con 0x7f4454101950 2023-07-20T02:51:50.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.322+0000 7f445a3ff700 1 -- 172.21.15.7:0/3171141306 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4454192d70 con 0x7f4454101950 2023-07-20T02:51:50.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.324+0000 7f44517fa700 1 -- 172.21.15.7:0/3171141306 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f4448020030 con 0x7f4454101950 2023-07-20T02:51:50.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.326+0000 7f44517fa700 1 --2- 172.21.15.7:0/3171141306 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f443c0a11d0 0x7f443c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:50.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.326+0000 7f44517fa700 1 -- 172.21.15.7:0/3171141306 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f44480cda80 con 0x7f4454101950 2023-07-20T02:51:50.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.326+0000 7f4453fff700 1 --2- 172.21.15.7:0/3171141306 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f443c0a11d0 0x7f443c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:50.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.327+0000 7f4453fff700 1 --2- 172.21.15.7:0/3171141306 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f443c0a11d0 0x7f443c0a3690 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f44541017b0 tx=0x7f4444009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:50.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.330+0000 7f44517fa700 1 -- 172.21.15.7:0/3171141306 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4448097010 con 0x7f4454101950 2023-07-20T02:51:50.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.556+0000 7f445a3ff700 1 -- 172.21.15.7:0/3171141306 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f445402c970 con 0x7f4454101950 2023-07-20T02:51:50.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.556+0000 7f44517fa700 1 -- 172.21.15.7:0/3171141306 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v87) v1 ==== 81+0+581 (secure 0 0 0) 0x7f444809a630 con 0x7f4454101950 2023-07-20T02:51:50.570 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/3892248290 2023-07-21T02:44:53.458649+0000 2023-07-20T02:51:50.570 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/2373058722 2023-07-21T02:44:53.458649+0000 2023-07-20T02:51:50.570 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6801/2926310425 2023-07-21T02:44:53.458649+0000 2023-07-20T02:51:50.571 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/4222569264 2023-07-21T02:44:25.804258+0000 2023-07-20T02:51:50.571 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/1357065986 2023-07-21T02:44:25.804258+0000 2023-07-20T02:51:50.571 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/884368977 2023-07-21T02:44:53.458649+0000 2023-07-20T02:51:50.571 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6801/1224472946 2023-07-21T02:44:25.804258+0000 2023-07-20T02:51:50.571 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/1459293748 2023-07-21T02:44:25.804258+0000 2023-07-20T02:51:50.571 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6800/1224472946 2023-07-21T02:44:25.804258+0000 2023-07-20T02:51:50.571 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6800/2926310425 2023-07-21T02:44:53.458649+0000 2023-07-20T02:51:50.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.558+0000 7f445a3ff700 1 -- 172.21.15.7:0/3171141306 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f443c0a11d0 msgr2=0x7f443c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:50.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.559+0000 7f445a3ff700 1 --2- 172.21.15.7:0/3171141306 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f443c0a11d0 0x7f443c0a3690 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f44541017b0 tx=0x7f4444009310 comp rx=0 tx=0).stop 2023-07-20T02:51:50.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.559+0000 7f445a3ff700 1 -- 172.21.15.7:0/3171141306 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4454101950 msgr2=0x7f4454198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:50.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.559+0000 7f445a3ff700 1 --2- 172.21.15.7:0/3171141306 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4454101950 0x7f4454198c40 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4448009230 tx=0x7f444800bf80 comp rx=0 tx=0).stop 2023-07-20T02:51:50.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.559+0000 7f445a3ff700 1 -- 172.21.15.7:0/3171141306 shutdown_connections 2023-07-20T02:51:50.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.559+0000 7f445a3ff700 1 --2- 172.21.15.7:0/3171141306 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44541022f0 0x7f445419fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:50.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.559+0000 7f445a3ff700 1 --2- 172.21.15.7:0/3171141306 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f4454100760 0x7f4454198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:50.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.559+0000 7f445a3ff700 1 --2- 172.21.15.7:0/3171141306 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f443c0a11d0 0x7f443c0a3690 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:50.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.559+0000 7f445a3ff700 1 --2- 172.21.15.7:0/3171141306 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4454101950 0x7f4454198c40 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:50.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.559+0000 7f445a3ff700 1 -- 172.21.15.7:0/3171141306 >> 172.21.15.7:0/3171141306 conn(0x7f44540fbcb0 msgr2=0x7f4454107330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:50.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.559+0000 7f445a3ff700 1 -- 172.21.15.7:0/3171141306 shutdown_connections 2023-07-20T02:51:50.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:50.559+0000 7f445a3ff700 1 -- 172.21.15.7:0/3171141306 wait complete. 2023-07-20T02:51:50.574 INFO:teuthology.orchestra.run.smithi007.stderr:listed 10 entries 2023-07-20T02:51:50.591 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi007.front.sepia.ceph.com... 2023-07-20T02:51:50.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T02:51:50.592 DEBUG:teuthology.orchestra.run.smithi007:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2023-07-20T02:51:50.659 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi007.stderr:ceph-fuse[163864]: fuse finished with error 0 and tester_r 0 2023-07-20T02:51:50.662 INFO:teuthology.orchestra.run:waiting for 300 2023-07-20T02:51:50.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:50 smithi158 ceph-mon[128925]: pgmap v285: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-07-20T02:51:50.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:50 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/3204411248' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-20T02:51:50.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:50 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/1010169436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-20T02:51:50.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:50 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/3171141306' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-20T02:51:50.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:50 smithi007 ceph-mon[126353]: pgmap v285: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-07-20T02:51:50.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:50 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/3204411248' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-20T02:51:50.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:50 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/1010169436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-20T02:51:50.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:50 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/3171141306' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-20T02:51:50.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:50 smithi188 ceph-mon[129553]: pgmap v285: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-07-20T02:51:50.965 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:50 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/3204411248' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-20T02:51:50.965 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:50 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/1010169436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-20T02:51:50.965 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:50 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/3171141306' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-20T02:51:52.891 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:52 smithi158 ceph-mon[128925]: pgmap v286: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-07-20T02:51:52.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:52 smithi007 ceph-mon[126353]: pgmap v286: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-07-20T02:51:52.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:52 smithi188 ceph-mon[129553]: pgmap v286: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-07-20T02:51:53.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:53 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2023-07-20T02:51:53.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:53 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]: dispatch 2023-07-20T02:51:53.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:53 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2023-07-20T02:51:53.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:53 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]: dispatch 2023-07-20T02:51:53.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:53 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2023-07-20T02:51:53.950 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:53 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2023-07-20T02:51:53.950 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:53 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2023-07-20T02:51:53.950 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:53 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2023-07-20T02:51:53.950 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:53 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]: dispatch 2023-07-20T02:51:53.950 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:53 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2023-07-20T02:51:53.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:53 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2023-07-20T02:51:53.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:53 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]: dispatch 2023-07-20T02:51:53.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:53 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2023-07-20T02:51:53.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:53 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]: dispatch 2023-07-20T02:51:53.965 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:53 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2023-07-20T02:51:53.965 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:53 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2023-07-20T02:51:53.965 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:53 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2023-07-20T02:51:53.965 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:53 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2023-07-20T02:51:53.965 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:53 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]: dispatch 2023-07-20T02:51:53.965 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:53 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2023-07-20T02:51:54.141 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:53 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2023-07-20T02:51:54.141 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:53 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]: dispatch 2023-07-20T02:51:54.141 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:53 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2023-07-20T02:51:54.142 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:53 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]: dispatch 2023-07-20T02:51:54.142 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:53 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2023-07-20T02:51:54.142 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:53 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2023-07-20T02:51:54.142 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:53 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2023-07-20T02:51:54.142 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:53 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2023-07-20T02:51:54.142 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:53 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]: dispatch 2023-07-20T02:51:54.143 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:53 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2023-07-20T02:51:54.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:54 smithi007 ceph-mon[126353]: pgmap v287: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-07-20T02:51:54.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:54 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2023-07-20T02:51:54.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:54 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]': finished 2023-07-20T02:51:54.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:54 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2023-07-20T02:51:54.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:54 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]': finished 2023-07-20T02:51:54.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:54 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2023-07-20T02:51:54.950 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:54 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2023-07-20T02:51:54.950 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:54 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2023-07-20T02:51:54.950 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:54 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2023-07-20T02:51:54.950 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:54 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]': finished 2023-07-20T02:51:54.951 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:54 smithi007 ceph-mon[126353]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2023-07-20T02:51:54.951 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:54 smithi007 ceph-mon[126353]: osdmap e88: 12 total, 12 up, 12 in 2023-07-20T02:51:54.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:54 smithi188 ceph-mon[129553]: pgmap v287: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-07-20T02:51:54.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:54 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2023-07-20T02:51:54.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:54 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]': finished 2023-07-20T02:51:54.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:54 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2023-07-20T02:51:54.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:54 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]': finished 2023-07-20T02:51:54.965 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:54 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2023-07-20T02:51:54.965 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:54 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2023-07-20T02:51:54.965 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:54 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2023-07-20T02:51:54.965 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:54 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2023-07-20T02:51:54.965 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:54 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]': finished 2023-07-20T02:51:54.965 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:54 smithi188 ceph-mon[129553]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2023-07-20T02:51:54.965 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:54 smithi188 ceph-mon[129553]: osdmap e88: 12 total, 12 up, 12 in 2023-07-20T02:51:55.141 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:54 smithi158 ceph-mon[128925]: pgmap v287: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-07-20T02:51:55.142 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:54 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2023-07-20T02:51:55.142 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:54 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]': finished 2023-07-20T02:51:55.142 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:54 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2023-07-20T02:51:55.142 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:54 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]': finished 2023-07-20T02:51:55.143 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:54 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2023-07-20T02:51:55.143 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:54 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2023-07-20T02:51:55.143 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:54 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2023-07-20T02:51:55.143 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:54 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2023-07-20T02:51:55.143 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:54 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]': finished 2023-07-20T02:51:55.143 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:54 smithi158 ceph-mon[128925]: from='mgr.14152 172.21.15.7:0/3670818081' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2023-07-20T02:51:55.144 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:54 smithi158 ceph-mon[128925]: osdmap e88: 12 total, 12 up, 12 in 2023-07-20T02:51:55.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:55 smithi007 ceph-mon[126353]: osdmap e89: 12 total, 12 up, 12 in 2023-07-20T02:51:55.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:55 smithi188 ceph-mon[129553]: osdmap e89: 12 total, 12 up, 12 in 2023-07-20T02:51:56.141 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:55 smithi158 ceph-mon[128925]: osdmap e89: 12 total, 12 up, 12 in 2023-07-20T02:51:56.764 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi007.front.sepia.ceph.com 2023-07-20T02:51:56.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-20T02:51:56.765 DEBUG:teuthology.orchestra.run.smithi007:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2023-07-20T02:51:56.832 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T02:51:56.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:56 smithi007 ceph-mon[126353]: pgmap v290: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s wr, 0 op/s 2023-07-20T02:51:56.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:56 smithi188 ceph-mon[129553]: pgmap v290: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s wr, 0 op/s 2023-07-20T02:51:57.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.028+0000 7fc1225ff700 1 -- 172.21.15.7:0/3377248023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc11c100770 msgr2=0x7fc11c100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:57.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.028+0000 7fc1225ff700 1 --2- 172.21.15.7:0/3377248023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc11c100770 0x7fc11c100b90 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc10c0099c0 tx=0x7fc10c009cd0 comp rx=0 tx=0).stop 2023-07-20T02:51:57.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.028+0000 7fc1225ff700 1 -- 172.21.15.7:0/3377248023 shutdown_connections 2023-07-20T02:51:57.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.028+0000 7fc1225ff700 1 --2- 172.21.15.7:0/3377248023 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc11c102300 0x7fc11c108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:57.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.028+0000 7fc1225ff700 1 --2- 172.21.15.7:0/3377248023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc11c100770 0x7fc11c100b90 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:57.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.028+0000 7fc1225ff700 1 --2- 172.21.15.7:0/3377248023 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc11c101960 0x7fc11c101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:57.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.028+0000 7fc1225ff700 1 -- 172.21.15.7:0/3377248023 >> 172.21.15.7:0/3377248023 conn(0x7fc11c0fbca0 msgr2=0x7fc11c0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:57.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.028+0000 7fc1225ff700 1 -- 172.21.15.7:0/3377248023 shutdown_connections 2023-07-20T02:51:57.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.028+0000 7fc1225ff700 1 -- 172.21.15.7:0/3377248023 wait complete. 2023-07-20T02:51:57.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.029+0000 7fc1225ff700 1 Processor -- start 2023-07-20T02:51:57.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.029+0000 7fc1225ff700 1 -- start start 2023-07-20T02:51:57.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.029+0000 7fc1225ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc11c100770 0x7fc11c196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:57.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.029+0000 7fc1225ff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc11c101960 0x7fc11c1969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:57.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.029+0000 7fc1225ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc11c102300 0x7fc11c19da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:57.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.029+0000 7fc1225ff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c19dff0 con 0x7fc11c102300 2023-07-20T02:51:57.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.029+0000 7fc1225ff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c19e130 con 0x7fc11c101960 2023-07-20T02:51:57.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.029+0000 7fc1225ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c19e2c0 con 0x7fc11c100770 2023-07-20T02:51:57.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.030+0000 7fc11b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc11c101960 0x7fc11c1969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:57.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.030+0000 7fc11b7fe700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc11c101960 0x7fc11c1969b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.7:50328/0 (socket says 172.21.15.7:50328) 2023-07-20T02:51:57.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.030+0000 7fc120b9c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc11c102300 0x7fc11c19da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:57.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.030+0000 7fc11b7fe700 1 -- 172.21.15.7:0/596810578 learned_addr learned my addr 172.21.15.7:0/596810578 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:57.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.030+0000 7fc11bfff700 1 --2- 172.21.15.7:0/596810578 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc11c100770 0x7fc11c196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:57.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.030+0000 7fc11b7fe700 1 -- 172.21.15.7:0/596810578 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc11c100770 msgr2=0x7fc11c196470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:57.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.030+0000 7fc11b7fe700 1 --2- 172.21.15.7:0/596810578 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc11c100770 0x7fc11c196470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:57.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.030+0000 7fc11b7fe700 1 -- 172.21.15.7:0/596810578 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc11c102300 msgr2=0x7fc11c19da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:57.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.030+0000 7fc11b7fe700 1 --2- 172.21.15.7:0/596810578 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc11c102300 0x7fc11c19da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:57.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.030+0000 7fc11b7fe700 1 -- 172.21.15.7:0/596810578 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc10c0095f0 con 0x7fc11c101960 2023-07-20T02:51:57.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.030+0000 7fc11b7fe700 1 --2- 172.21.15.7:0/596810578 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc11c101960 0x7fc11c1969b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc11000d950 tx=0x7fc11000dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:57.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.031+0000 7fc1197fa700 1 -- 172.21.15.7:0/596810578 <== mon.2 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11000f890 con 0x7fc11c101960 2023-07-20T02:51:57.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.031+0000 7fc1225ff700 1 -- 172.21.15.7:0/596810578 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc11c19e5a0 con 0x7fc11c101960 2023-07-20T02:51:57.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.031+0000 7fc1197fa700 1 -- 172.21.15.7:0/596810578 <== mon.2 v2:172.21.15.158:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc110010460 con 0x7fc11c101960 2023-07-20T02:51:57.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.031+0000 7fc1197fa700 1 -- 172.21.15.7:0/596810578 <== mon.2 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11000e610 con 0x7fc11c101960 2023-07-20T02:51:57.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.031+0000 7fc1225ff700 1 -- 172.21.15.7:0/596810578 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc11c19eab0 con 0x7fc11c101960 2023-07-20T02:51:57.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.032+0000 7fc1225ff700 1 -- 172.21.15.7:0/596810578 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc11c10bfe0 con 0x7fc11c101960 2023-07-20T02:51:57.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.034+0000 7fc1197fa700 1 -- 172.21.15.7:0/596810578 <== mon.2 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fc1100108d0 con 0x7fc11c101960 2023-07-20T02:51:57.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.035+0000 7fc1197fa700 1 --2- 172.21.15.7:0/596810578 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fc1040a11a0 0x7fc1040a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:57.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.035+0000 7fc1197fa700 1 -- 172.21.15.7:0/596810578 <== mon.2 v2:172.21.15.158:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc1100cca20 con 0x7fc11c101960 2023-07-20T02:51:57.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.035+0000 7fc11bfff700 1 --2- 172.21.15.7:0/596810578 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fc1040a11a0 0x7fc1040a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:57.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.036+0000 7fc11bfff700 1 --2- 172.21.15.7:0/596810578 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fc1040a11a0 0x7fc1040a3660 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc10c00b580 tx=0x7fc10c009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:57.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.037+0000 7fc1197fa700 1 -- 172.21.15.7:0/596810578 <== mon.2 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc110095e00 con 0x7fc11c101960 2023-07-20T02:51:57.141 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:56 smithi158 ceph-mon[128925]: pgmap v290: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s wr, 0 op/s 2023-07-20T02:51:57.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.251+0000 7fc1225ff700 1 -- 172.21.15.7:0/596810578 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc11c10c1d0 con 0x7fc11c101960 2023-07-20T02:51:57.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.254+0000 7fc1197fa700 1 -- 172.21.15.7:0/596810578 <== mon.2 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24396 (secure 0 0 0) 0x7fc110016090 con 0x7fc11c101960 2023-07-20T02:51:57.268 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:51:57.268 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":89,"fsid":"1b04643c-26a7-11ee-9b34-001a4aab830c","created":"2023-07-20T02:43:56.544719+0000","modified":"2023-07-20T02:51:54.629288+0000","last_up_change":"2023-07-20T02:50:40.126926+0000","last_in_change":"2023-07-20T02:50:21.875020+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T02:47:12.055539+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-20T02:51:26.617961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-20T02:51:28.081016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-20T02:51:31.658088+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ad6c6339-6e23-4c21-9fde-2dbecdafdb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6803","nonce":3322992091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6805","nonce":3322992091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6809","nonce":3322992091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6807","nonce":3322992091}]},"public_addr":"172.21.15.7:6803/3322992091","cluster_addr":"172.21.15.7:6805/3322992091","heartbeat_back_addr":"172.21.15.7:6809/3322992091","heartbeat_front_addr":"172.21.15.7:6807/3322992091","state":["exists","up"]},{"osd":1,"uuid":"cad207b3-a1ea-4f07-bd94-9b11fd573a69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6801","nonce":2796771612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6803","nonce":2796771612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6807","nonce":2796771612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6805","nonce":2796771612}]},"public_addr":"172.21.15.158:6801/2796771612","cluster_addr":"172.21.15.158:6803/2796771612","heartbeat_back_addr":"172.21.15.158:6807/2796771612","heartbeat_front_addr":"172.21.15.158:6805/2796771612","state":["exists","up"]},{"osd":2,"uuid":"2c777fe6-d003-49d0-af37-f63fca37ff06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6801","nonce":2928307258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6803","nonce":2928307258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6807","nonce":2928307258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6805","nonce":2928307258}]},"public_addr":"172.21.15.188:6801/2928307258","cluster_addr":"172.21.15.188:6803/2928307258","heartbeat_back_addr":"172.21.15.188:6807/2928307258","heartbeat_front_addr":"172.21.15.188:6805/2928307258","state":["exists","up"]},{"osd":3,"uuid":"59e29965-018f-4eb6-adbd-1a9e1df84245","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6811","nonce":2192953827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6813","nonce":2192953827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6817","nonce":2192953827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6815","nonce":2192953827}]},"public_addr":"172.21.15.7:6811/2192953827","cluster_addr":"172.21.15.7:6813/2192953827","heartbeat_back_addr":"172.21.15.7:6817/2192953827","heartbeat_front_addr":"172.21.15.7:6815/2192953827","state":["exists","up"]},{"osd":4,"uuid":"e2453384-3b09-4f2a-bea7-172d8fc466d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6809","nonce":3330591132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6810","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6811","nonce":3330591132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6814","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6815","nonce":3330591132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6813","nonce":3330591132}]},"public_addr":"172.21.15.158:6809/3330591132","cluster_addr":"172.21.15.158:6811/3330591132","heartbeat_back_addr":"172.21.15.158:6815/3330591132","heartbeat_front_addr":"172.21.15.158:6813/3330591132","state":["exists","up"]},{"osd":5,"uuid":"c3c04c13-0ce5-440d-9e62-0e301ad9f771","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6809","nonce":3142012805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6811","nonce":3142012805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6815","nonce":3142012805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6813","nonce":3142012805}]},"public_addr":"172.21.15.188:6809/3142012805","cluster_addr":"172.21.15.188:6811/3142012805","heartbeat_back_addr":"172.21.15.188:6815/3142012805","heartbeat_front_addr":"172.21.15.188:6813/3142012805","state":["exists","up"]},{"osd":6,"uuid":"82ef2d33-1045-41f1-bf69-b03404bcace8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6819","nonce":859560576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6821","nonce":859560576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6825","nonce":859560576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6823","nonce":859560576}]},"public_addr":"172.21.15.7:6819/859560576","cluster_addr":"172.21.15.7:6821/859560576","heartbeat_back_addr":"172.21.15.7:6825/859560576","heartbeat_front_addr":"172.21.15.7:6823/859560576","state":["exists","up"]},{"osd":7,"uuid":"020ec859-3fc2-4da2-a0dc-10530a4a366c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6817","nonce":2578758635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6819","nonce":2578758635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6822","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6823","nonce":2578758635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6821","nonce":2578758635}]},"public_addr":"172.21.15.158:6817/2578758635","cluster_addr":"172.21.15.158:6819/2578758635","heartbeat_back_addr":"172.21.15.158:6823/2578758635","heartbeat_front_addr":"172.21.15.158:6821/2578758635","state":["exists","up"]},{"osd":8,"uuid":"4081496a-45ee-4b25-8f17-a23665521187","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6817","nonce":3020824843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6819","nonce":3020824843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6823","nonce":3020824843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6821","nonce":3020824843}]},"public_addr":"172.21.15.188:6817/3020824843","cluster_addr":"172.21.15.188:6819/3020824843","heartbeat_back_addr":"172.21.15.188:6823/3020824843","heartbeat_front_addr":"172.21.15.188:6821/3020824843","state":["exists","up"]},{"osd":9,"uuid":"723580ae-efe3-4d86-95fd-d7a4c0631425","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6827","nonce":1612464825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6829","nonce":1612464825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6833","nonce":1612464825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6831","nonce":1612464825}]},"public_addr":"172.21.15.7:6827/1612464825","cluster_addr":"172.21.15.7:6829/1612464825","heartbeat_back_addr":"172.21.15.7:6833/1612464825","heartbeat_front_addr":"172.21.15.7:6831/1612464825","state":["exists","up"]},{"osd":10,"uuid":"c2651548-fe46-4d2d-b911-278036412ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6825","nonce":1653166210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6827","nonce":1653166210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6831","nonce":1653166210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6829","nonce":1653166210}]},"public_addr":"172.21.15.158:6825/1653166210","cluster_addr":"172.21.15.158:6827/1653166210","heartbeat_back_addr":"172.21.15.158:6831/1653166210","heartbeat_front_addr":"172.21.15.158:6829/1653166210","state":["exists","up"]},{"osd":11,"uuid":"f7262bcb-4243-403a-a321-fc2bdf1d4d25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6825","nonce":1230196525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6827","nonce":1230196525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6831","nonce":1230196525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6829","nonce":1230196525}]},"public_addr":"172.21.15.188:6825/1230196525","cluster_addr":"172.21.15.188:6827/1230196525","heartbeat_back_addr":"172.21.15.188:6831/1230196525","heartbeat_front_addr":"172.21.15.188:6829/1230196525","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:46:26.803832+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:46:48.726253+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:47:09.744123+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:47:36.907588+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:47:57.696685+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:48:19.120806+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:48:45.313043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:49:06.686384+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:49:28.475078+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:49:54.984033+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:50:16.541259+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:50:38.221746+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/3892248290":"2023-07-21T02:44:53.458649+0000","172.21.15.7:0/2373058722":"2023-07-21T02:44:53.458649+0000","172.21.15.7:6801/2926310425":"2023-07-21T02:44:53.458649+0000","172.21.15.7:0/4222569264":"2023-07-21T02:44:25.804258+0000","172.21.15.7:0/1357065986":"2023-07-21T02:44:25.804258+0000","172.21.15.7:0/884368977":"2023-07-21T02:44:53.458649+0000","172.21.15.7:6801/1224472946":"2023-07-21T02:44:25.804258+0000","172.21.15.7:0/1459293748":"2023-07-21T02:44:25.804258+0000","172.21.15.7:6800/1224472946":"2023-07-21T02:44:25.804258+0000","172.21.15.7:6800/2926310425":"2023-07-21T02:44:53.458649+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T02:51:57.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.256+0000 7fc1225ff700 1 -- 172.21.15.7:0/596810578 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fc1040a11a0 msgr2=0x7fc1040a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:57.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.256+0000 7fc1225ff700 1 --2- 172.21.15.7:0/596810578 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fc1040a11a0 0x7fc1040a3660 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc10c00b580 tx=0x7fc10c009f10 comp rx=0 tx=0).stop 2023-07-20T02:51:57.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.256+0000 7fc1225ff700 1 -- 172.21.15.7:0/596810578 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc11c101960 msgr2=0x7fc11c1969b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:57.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.256+0000 7fc1225ff700 1 --2- 172.21.15.7:0/596810578 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc11c101960 0x7fc11c1969b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc11000d950 tx=0x7fc11000dc60 comp rx=0 tx=0).stop 2023-07-20T02:51:57.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.256+0000 7fc1225ff700 1 -- 172.21.15.7:0/596810578 shutdown_connections 2023-07-20T02:51:57.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.256+0000 7fc1225ff700 1 --2- 172.21.15.7:0/596810578 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fc1040a11a0 0x7fc1040a3660 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:57.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.256+0000 7fc1225ff700 1 --2- 172.21.15.7:0/596810578 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc11c102300 0x7fc11c19da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:57.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.256+0000 7fc1225ff700 1 --2- 172.21.15.7:0/596810578 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc11c100770 0x7fc11c196470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:57.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.256+0000 7fc1225ff700 1 --2- 172.21.15.7:0/596810578 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fc11c101960 0x7fc11c1969b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:57.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.256+0000 7fc1225ff700 1 -- 172.21.15.7:0/596810578 >> 172.21.15.7:0/596810578 conn(0x7fc11c0fbca0 msgr2=0x7fc11c0fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:57.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.256+0000 7fc1225ff700 1 -- 172.21.15.7:0/596810578 shutdown_connections 2023-07-20T02:51:57.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.256+0000 7fc1225ff700 1 -- 172.21.15.7:0/596810578 wait complete. 2023-07-20T02:51:57.289 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T02:51:57.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.479+0000 7f120bdff700 1 -- 172.21.15.7:0/1964261465 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1204102300 msgr2=0x7f1204108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:57.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.479+0000 7f120bdff700 1 --2- 172.21.15.7:0/1964261465 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1204102300 0x7f1204108bb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f11fc00b0d0 tx=0x7f11fc00b3e0 comp rx=0 tx=0).stop 2023-07-20T02:51:57.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.479+0000 7f120bdff700 1 -- 172.21.15.7:0/1964261465 shutdown_connections 2023-07-20T02:51:57.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.479+0000 7f120bdff700 1 --2- 172.21.15.7:0/1964261465 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1204102300 0x7f1204108bb0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:57.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.479+0000 7f120bdff700 1 --2- 172.21.15.7:0/1964261465 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1204100770 0x7f1204100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:57.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.479+0000 7f120bdff700 1 --2- 172.21.15.7:0/1964261465 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1204101960 0x7f1204101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:57.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.479+0000 7f120bdff700 1 -- 172.21.15.7:0/1964261465 >> 172.21.15.7:0/1964261465 conn(0x7f12040fbca0 msgr2=0x7f12040fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:57.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.479+0000 7f120bdff700 1 -- 172.21.15.7:0/1964261465 shutdown_connections 2023-07-20T02:51:57.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.479+0000 7f120bdff700 1 -- 172.21.15.7:0/1964261465 wait complete. 2023-07-20T02:51:57.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.480+0000 7f120bdff700 1 Processor -- start 2023-07-20T02:51:57.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.480+0000 7f120bdff700 1 -- start start 2023-07-20T02:51:57.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.480+0000 7f120bdff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1204100770 0x7f1204198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:57.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.480+0000 7f120bdff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1204101960 0x7f1204198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:57.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.480+0000 7f120bdff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1204102300 0x7f120419fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:57.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.480+0000 7f120bdff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12041a02c0 con 0x7f1204100770 2023-07-20T02:51:57.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.480+0000 7f120bdff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12041a0400 con 0x7f1204102300 2023-07-20T02:51:57.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.480+0000 7f120bdff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12041a0590 con 0x7f1204101960 2023-07-20T02:51:57.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.481+0000 7f1209b9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1204100770 0x7f1204198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:57.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.481+0000 7f1209b9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1204100770 0x7f1204198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46644/0 (socket says 172.21.15.7:46644) 2023-07-20T02:51:57.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.481+0000 7f120a39c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1204102300 0x7f120419fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:57.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.481+0000 7f1209b9b700 1 -- 172.21.15.7:0/1771576180 learned_addr learned my addr 172.21.15.7:0/1771576180 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:57.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.481+0000 7f120939a700 1 --2- 172.21.15.7:0/1771576180 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1204101960 0x7f1204198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:57.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.481+0000 7f1209b9b700 1 -- 172.21.15.7:0/1771576180 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1204101960 msgr2=0x7f1204198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:57.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.481+0000 7f1209b9b700 1 --2- 172.21.15.7:0/1771576180 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1204101960 0x7f1204198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:57.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.481+0000 7f1209b9b700 1 -- 172.21.15.7:0/1771576180 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1204102300 msgr2=0x7f120419fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:57.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.481+0000 7f1209b9b700 1 --2- 172.21.15.7:0/1771576180 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1204102300 0x7f120419fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:57.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.481+0000 7f1209b9b700 1 -- 172.21.15.7:0/1771576180 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11fc009d00 con 0x7f1204100770 2023-07-20T02:51:57.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.481+0000 7f120a39c700 1 --2- 172.21.15.7:0/1771576180 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1204102300 0x7f120419fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T02:51:57.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.481+0000 7f1209b9b700 1 --2- 172.21.15.7:0/1771576180 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1204100770 0x7f1204198740 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f120000e970 tx=0x7f120000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:57.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.481+0000 7f11faffd700 1 -- 172.21.15.7:0/1771576180 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1200009790 con 0x7f1204100770 2023-07-20T02:51:57.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.482+0000 7f11faffd700 1 -- 172.21.15.7:0/1771576180 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f120000c7e0 con 0x7f1204100770 2023-07-20T02:51:57.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.482+0000 7f11faffd700 1 -- 172.21.15.7:0/1771576180 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1200003c70 con 0x7f1204100770 2023-07-20T02:51:57.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.482+0000 7f120bdff700 1 -- 172.21.15.7:0/1771576180 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12041a0870 con 0x7f1204100770 2023-07-20T02:51:57.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.482+0000 7f120bdff700 1 -- 172.21.15.7:0/1771576180 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12041a0d30 con 0x7f1204100770 2023-07-20T02:51:57.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.483+0000 7f120bdff700 1 -- 172.21.15.7:0/1771576180 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f120410bfe0 con 0x7f1204100770 2023-07-20T02:51:57.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.485+0000 7f11faffd700 1 -- 172.21.15.7:0/1771576180 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f120000cc50 con 0x7f1204100770 2023-07-20T02:51:57.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.486+0000 7f11faffd700 1 --2- 172.21.15.7:0/1771576180 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f11f00a11a0 0x7f11f00a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:57.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.486+0000 7f11faffd700 1 -- 172.21.15.7:0/1771576180 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f1200014070 con 0x7f1204100770 2023-07-20T02:51:57.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.486+0000 7f120939a700 1 --2- 172.21.15.7:0/1771576180 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f11f00a11a0 0x7f11f00a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:57.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.487+0000 7f120939a700 1 --2- 172.21.15.7:0/1771576180 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f11f00a11a0 0x7f11f00a3660 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f11f4009c60 tx=0x7f11f4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:57.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.490+0000 7f11faffd700 1 -- 172.21.15.7:0/1771576180 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f12000961f0 con 0x7f1204100770 2023-07-20T02:51:57.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.699+0000 7f120bdff700 1 -- 172.21.15.7:0/1771576180 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f120410c200 con 0x7f1204100770 2023-07-20T02:51:57.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.701+0000 7f11faffd700 1 -- 172.21.15.7:0/1771576180 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24396 (secure 0 0 0) 0x7f1200003dd0 con 0x7f1204100770 2023-07-20T02:51:57.715 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-07-20T02:51:57.715 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":89,"fsid":"1b04643c-26a7-11ee-9b34-001a4aab830c","created":"2023-07-20T02:43:56.544719+0000","modified":"2023-07-20T02:51:54.629288+0000","last_up_change":"2023-07-20T02:50:40.126926+0000","last_in_change":"2023-07-20T02:50:21.875020+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T02:47:12.055539+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-20T02:51:26.617961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-20T02:51:28.081016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-20T02:51:31.658088+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ad6c6339-6e23-4c21-9fde-2dbecdafdb17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6803","nonce":3322992091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6805","nonce":3322992091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6809","nonce":3322992091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3322992091},{"type":"v1","addr":"172.21.15.7:6807","nonce":3322992091}]},"public_addr":"172.21.15.7:6803/3322992091","cluster_addr":"172.21.15.7:6805/3322992091","heartbeat_back_addr":"172.21.15.7:6809/3322992091","heartbeat_front_addr":"172.21.15.7:6807/3322992091","state":["exists","up"]},{"osd":1,"uuid":"cad207b3-a1ea-4f07-bd94-9b11fd573a69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6801","nonce":2796771612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6803","nonce":2796771612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6807","nonce":2796771612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2796771612},{"type":"v1","addr":"172.21.15.158:6805","nonce":2796771612}]},"public_addr":"172.21.15.158:6801/2796771612","cluster_addr":"172.21.15.158:6803/2796771612","heartbeat_back_addr":"172.21.15.158:6807/2796771612","heartbeat_front_addr":"172.21.15.158:6805/2796771612","state":["exists","up"]},{"osd":2,"uuid":"2c777fe6-d003-49d0-af37-f63fca37ff06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6801","nonce":2928307258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6803","nonce":2928307258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6807","nonce":2928307258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2928307258},{"type":"v1","addr":"172.21.15.188:6805","nonce":2928307258}]},"public_addr":"172.21.15.188:6801/2928307258","cluster_addr":"172.21.15.188:6803/2928307258","heartbeat_back_addr":"172.21.15.188:6807/2928307258","heartbeat_front_addr":"172.21.15.188:6805/2928307258","state":["exists","up"]},{"osd":3,"uuid":"59e29965-018f-4eb6-adbd-1a9e1df84245","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6811","nonce":2192953827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6813","nonce":2192953827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6817","nonce":2192953827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2192953827},{"type":"v1","addr":"172.21.15.7:6815","nonce":2192953827}]},"public_addr":"172.21.15.7:6811/2192953827","cluster_addr":"172.21.15.7:6813/2192953827","heartbeat_back_addr":"172.21.15.7:6817/2192953827","heartbeat_front_addr":"172.21.15.7:6815/2192953827","state":["exists","up"]},{"osd":4,"uuid":"e2453384-3b09-4f2a-bea7-172d8fc466d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6809","nonce":3330591132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6810","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6811","nonce":3330591132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6814","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6815","nonce":3330591132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":3330591132},{"type":"v1","addr":"172.21.15.158:6813","nonce":3330591132}]},"public_addr":"172.21.15.158:6809/3330591132","cluster_addr":"172.21.15.158:6811/3330591132","heartbeat_back_addr":"172.21.15.158:6815/3330591132","heartbeat_front_addr":"172.21.15.158:6813/3330591132","state":["exists","up"]},{"osd":5,"uuid":"c3c04c13-0ce5-440d-9e62-0e301ad9f771","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6809","nonce":3142012805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6811","nonce":3142012805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6815","nonce":3142012805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":3142012805},{"type":"v1","addr":"172.21.15.188:6813","nonce":3142012805}]},"public_addr":"172.21.15.188:6809/3142012805","cluster_addr":"172.21.15.188:6811/3142012805","heartbeat_back_addr":"172.21.15.188:6815/3142012805","heartbeat_front_addr":"172.21.15.188:6813/3142012805","state":["exists","up"]},{"osd":6,"uuid":"82ef2d33-1045-41f1-bf69-b03404bcace8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6819","nonce":859560576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6821","nonce":859560576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6825","nonce":859560576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":859560576},{"type":"v1","addr":"172.21.15.7:6823","nonce":859560576}]},"public_addr":"172.21.15.7:6819/859560576","cluster_addr":"172.21.15.7:6821/859560576","heartbeat_back_addr":"172.21.15.7:6825/859560576","heartbeat_front_addr":"172.21.15.7:6823/859560576","state":["exists","up"]},{"osd":7,"uuid":"020ec859-3fc2-4da2-a0dc-10530a4a366c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6817","nonce":2578758635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6819","nonce":2578758635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6822","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6823","nonce":2578758635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":2578758635},{"type":"v1","addr":"172.21.15.158:6821","nonce":2578758635}]},"public_addr":"172.21.15.158:6817/2578758635","cluster_addr":"172.21.15.158:6819/2578758635","heartbeat_back_addr":"172.21.15.158:6823/2578758635","heartbeat_front_addr":"172.21.15.158:6821/2578758635","state":["exists","up"]},{"osd":8,"uuid":"4081496a-45ee-4b25-8f17-a23665521187","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6817","nonce":3020824843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6819","nonce":3020824843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6823","nonce":3020824843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3020824843},{"type":"v1","addr":"172.21.15.188:6821","nonce":3020824843}]},"public_addr":"172.21.15.188:6817/3020824843","cluster_addr":"172.21.15.188:6819/3020824843","heartbeat_back_addr":"172.21.15.188:6823/3020824843","heartbeat_front_addr":"172.21.15.188:6821/3020824843","state":["exists","up"]},{"osd":9,"uuid":"723580ae-efe3-4d86-95fd-d7a4c0631425","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6827","nonce":1612464825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6829","nonce":1612464825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6833","nonce":1612464825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1612464825},{"type":"v1","addr":"172.21.15.7:6831","nonce":1612464825}]},"public_addr":"172.21.15.7:6827/1612464825","cluster_addr":"172.21.15.7:6829/1612464825","heartbeat_back_addr":"172.21.15.7:6833/1612464825","heartbeat_front_addr":"172.21.15.7:6831/1612464825","state":["exists","up"]},{"osd":10,"uuid":"c2651548-fe46-4d2d-b911-278036412ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6825","nonce":1653166210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6826","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6827","nonce":1653166210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6830","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6831","nonce":1653166210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6828","nonce":1653166210},{"type":"v1","addr":"172.21.15.158:6829","nonce":1653166210}]},"public_addr":"172.21.15.158:6825/1653166210","cluster_addr":"172.21.15.158:6827/1653166210","heartbeat_back_addr":"172.21.15.158:6831/1653166210","heartbeat_front_addr":"172.21.15.158:6829/1653166210","state":["exists","up"]},{"osd":11,"uuid":"f7262bcb-4243-403a-a321-fc2bdf1d4d25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6825","nonce":1230196525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6827","nonce":1230196525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6831","nonce":1230196525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":1230196525},{"type":"v1","addr":"172.21.15.188:6829","nonce":1230196525}]},"public_addr":"172.21.15.188:6825/1230196525","cluster_addr":"172.21.15.188:6827/1230196525","heartbeat_back_addr":"172.21.15.188:6831/1230196525","heartbeat_front_addr":"172.21.15.188:6829/1230196525","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:46:26.803832+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:46:48.726253+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:47:09.744123+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:47:36.907588+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:47:57.696685+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:48:19.120806+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:48:45.313043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:49:06.686384+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:49:28.475078+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:49:54.984033+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:50:16.541259+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T02:50:38.221746+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/3892248290":"2023-07-21T02:44:53.458649+0000","172.21.15.7:0/2373058722":"2023-07-21T02:44:53.458649+0000","172.21.15.7:6801/2926310425":"2023-07-21T02:44:53.458649+0000","172.21.15.7:0/4222569264":"2023-07-21T02:44:25.804258+0000","172.21.15.7:0/1357065986":"2023-07-21T02:44:25.804258+0000","172.21.15.7:0/884368977":"2023-07-21T02:44:53.458649+0000","172.21.15.7:6801/1224472946":"2023-07-21T02:44:25.804258+0000","172.21.15.7:0/1459293748":"2023-07-21T02:44:25.804258+0000","172.21.15.7:6800/1224472946":"2023-07-21T02:44:25.804258+0000","172.21.15.7:6800/2926310425":"2023-07-21T02:44:53.458649+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T02:51:57.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.703+0000 7f120bdff700 1 -- 172.21.15.7:0/1771576180 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f11f00a11a0 msgr2=0x7f11f00a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:57.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.703+0000 7f120bdff700 1 --2- 172.21.15.7:0/1771576180 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f11f00a11a0 0x7f11f00a3660 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f11f4009c60 tx=0x7f11f4009260 comp rx=0 tx=0).stop 2023-07-20T02:51:57.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.703+0000 7f120bdff700 1 -- 172.21.15.7:0/1771576180 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1204100770 msgr2=0x7f1204198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:57.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.703+0000 7f120bdff700 1 --2- 172.21.15.7:0/1771576180 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1204100770 0x7f1204198740 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f120000e970 tx=0x7f120000ec80 comp rx=0 tx=0).stop 2023-07-20T02:51:57.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.703+0000 7f120bdff700 1 -- 172.21.15.7:0/1771576180 shutdown_connections 2023-07-20T02:51:57.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.703+0000 7f120bdff700 1 --2- 172.21.15.7:0/1771576180 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f1204102300 0x7f120419fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:57.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.703+0000 7f120bdff700 1 --2- 172.21.15.7:0/1771576180 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f11f00a11a0 0x7f11f00a3660 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:57.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.703+0000 7f120bdff700 1 --2- 172.21.15.7:0/1771576180 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1204100770 0x7f1204198740 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:57.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.703+0000 7f120bdff700 1 --2- 172.21.15.7:0/1771576180 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1204101960 0x7f1204198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:57.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.703+0000 7f120bdff700 1 -- 172.21.15.7:0/1771576180 >> 172.21.15.7:0/1771576180 conn(0x7f12040fbca0 msgr2=0x7f12041071c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:57.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.703+0000 7f120bdff700 1 -- 172.21.15.7:0/1771576180 shutdown_connections 2023-07-20T02:51:57.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.703+0000 7f120bdff700 1 -- 172.21.15.7:0/1771576180 wait complete. 2023-07-20T02:51:57.736 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-07-20T02:51:57.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.929+0000 7fb7ccbff700 1 -- 172.21.15.7:0/2921364791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7c8101950 msgr2=0x7fb7c8101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:57.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.929+0000 7fb7ccbff700 1 --2- 172.21.15.7:0/2921364791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7c8101950 0x7fb7c8101db0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc009990 tx=0x7fb7bc009ca0 comp rx=0 tx=0).stop 2023-07-20T02:51:57.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.929+0000 7fb7ccbff700 1 -- 172.21.15.7:0/2921364791 shutdown_connections 2023-07-20T02:51:57.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.929+0000 7fb7ccbff700 1 --2- 172.21.15.7:0/2921364791 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7c81022f0 0x7fb7c8108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:57.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.929+0000 7fb7ccbff700 1 --2- 172.21.15.7:0/2921364791 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7c8100760 0x7fb7c8100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:57.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.929+0000 7fb7ccbff700 1 --2- 172.21.15.7:0/2921364791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7c8101950 0x7fb7c8101db0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:57.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.929+0000 7fb7ccbff700 1 -- 172.21.15.7:0/2921364791 >> 172.21.15.7:0/2921364791 conn(0x7fb7c80fbcb0 msgr2=0x7fb7c80fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:57.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.929+0000 7fb7ccbff700 1 -- 172.21.15.7:0/2921364791 shutdown_connections 2023-07-20T02:51:57.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.929+0000 7fb7ccbff700 1 -- 172.21.15.7:0/2921364791 wait complete. 2023-07-20T02:51:57.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.930+0000 7fb7ccbff700 1 Processor -- start 2023-07-20T02:51:57.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.930+0000 7fb7ccbff700 1 -- start start 2023-07-20T02:51:57.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.930+0000 7fb7ccbff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7c8100760 0x7fb7c8196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:57.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.930+0000 7fb7ccbff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7c8101950 0x7fb7c8196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:57.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.930+0000 7fb7ccbff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7c81022f0 0x7fb7c819da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:57.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.930+0000 7fb7ccbff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c819dfd0 con 0x7fb7c8101950 2023-07-20T02:51:57.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.930+0000 7fb7ccbff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c819e110 con 0x7fb7c8100760 2023-07-20T02:51:57.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.930+0000 7fb7ccbff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c819e2a0 con 0x7fb7c81022f0 2023-07-20T02:51:57.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.931+0000 7fb7c6d9d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7c81022f0 0x7fb7c819da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:57.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.931+0000 7fb7c659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7c8100760 0x7fb7c8196450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:57.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.931+0000 7fb7c5d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7c8101950 0x7fb7c8196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:57.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.931+0000 7fb7c659c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7c8100760 0x7fb7c8196450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.7:50358/0 (socket says 172.21.15.7:50358) 2023-07-20T02:51:57.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.931+0000 7fb7c6d9d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7c81022f0 0x7fb7c819da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.7:39246/0 (socket says 172.21.15.7:39246) 2023-07-20T02:51:57.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.931+0000 7fb7c659c700 1 -- 172.21.15.7:0/1902595222 learned_addr learned my addr 172.21.15.7:0/1902595222 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:57.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.931+0000 7fb7c659c700 1 -- 172.21.15.7:0/1902595222 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7c81022f0 msgr2=0x7fb7c819da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:57.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.931+0000 7fb7c659c700 1 --2- 172.21.15.7:0/1902595222 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7c81022f0 0x7fb7c819da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:57.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.931+0000 7fb7c659c700 1 -- 172.21.15.7:0/1902595222 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7c8101950 msgr2=0x7fb7c8196990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:57.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.931+0000 7fb7c659c700 1 --2- 172.21.15.7:0/1902595222 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7c8101950 0x7fb7c8196990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:57.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.931+0000 7fb7c659c700 1 -- 172.21.15.7:0/1902595222 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7bc0095f0 con 0x7fb7c8100760 2023-07-20T02:51:57.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.931+0000 7fb7c5d9b700 1 --2- 172.21.15.7:0/1902595222 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7c8101950 0x7fb7c8196990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T02:51:57.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.931+0000 7fb7c659c700 1 --2- 172.21.15.7:0/1902595222 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7c8100760 0x7fb7c8196450 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb7b800e970 tx=0x7fb7b800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:57.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.932+0000 7fb7b77fe700 1 -- 172.21.15.7:0/1902595222 <== mon.2 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b8009790 con 0x7fb7c8100760 2023-07-20T02:51:57.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.932+0000 7fb7ccbff700 1 -- 172.21.15.7:0/1902595222 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7c819e580 con 0x7fb7c8100760 2023-07-20T02:51:57.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.932+0000 7fb7b77fe700 1 -- 172.21.15.7:0/1902595222 <== mon.2 v2:172.21.15.158:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb7b800c7e0 con 0x7fb7c8100760 2023-07-20T02:51:57.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.932+0000 7fb7b77fe700 1 -- 172.21.15.7:0/1902595222 <== mon.2 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b8003c70 con 0x7fb7c8100760 2023-07-20T02:51:57.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.932+0000 7fb7ccbff700 1 -- 172.21.15.7:0/1902595222 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7c819eae0 con 0x7fb7c8100760 2023-07-20T02:51:57.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.933+0000 7fb7ccbff700 1 -- 172.21.15.7:0/1902595222 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7c8107710 con 0x7fb7c8100760 2023-07-20T02:51:57.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.935+0000 7fb7b77fe700 1 -- 172.21.15.7:0/1902595222 <== mon.2 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fb7b800cc50 con 0x7fb7c8100760 2023-07-20T02:51:57.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.936+0000 7fb7b77fe700 1 --2- 172.21.15.7:0/1902595222 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fb7b00a1150 0x7fb7b00a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:57.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.936+0000 7fb7b77fe700 1 -- 172.21.15.7:0/1902595222 <== mon.2 v2:172.21.15.158:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fb7b8014070 con 0x7fb7c8100760 2023-07-20T02:51:57.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.937+0000 7fb7c5d9b700 1 --2- 172.21.15.7:0/1902595222 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fb7b00a1150 0x7fb7b00a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:57.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.937+0000 7fb7c5d9b700 1 --2- 172.21.15.7:0/1902595222 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fb7b00a1150 0x7fb7b00a3610 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00b2e0 tx=0x7fb7bc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:57.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:57.938+0000 7fb7b77fe700 1 -- 172.21.15.7:0/1902595222 <== mon.2 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb7b8096370 con 0x7fb7c8100760 2023-07-20T02:51:58.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:58.195+0000 7fb7ccbff700 1 -- 172.21.15.7:0/1902595222 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fb7c802cc00 con 0x7fb7c8100760 2023-07-20T02:51:58.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:58 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/596810578' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:51:58.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:58 smithi007 ceph-mon[126353]: pgmap v291: 129 pgs: 129 active+clean; 581 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 0 op/s 2023-07-20T02:51:58.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:58 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/1771576180' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:51:58.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:58 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/596810578' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:51:58.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:58 smithi188 ceph-mon[129553]: pgmap v291: 129 pgs: 129 active+clean; 581 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 0 op/s 2023-07-20T02:51:58.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:58 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/1771576180' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:51:58.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:58 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/596810578' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:51:58.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:58 smithi158 ceph-mon[128925]: pgmap v291: 129 pgs: 129 active+clean; 581 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 0 op/s 2023-07-20T02:51:58.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:58 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/1771576180' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T02:51:59.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.155+0000 7fb7b77fe700 1 -- 172.21.15.7:0/1902595222 <== mon.2 v2:172.21.15.158:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb7b8099990 con 0x7fb7c8100760 2023-07-20T02:51:59.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.158+0000 7fb7ccbff700 1 -- 172.21.15.7:0/1902595222 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fb7b00a1150 msgr2=0x7fb7b00a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:59.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.158+0000 7fb7ccbff700 1 --2- 172.21.15.7:0/1902595222 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fb7b00a1150 0x7fb7b00a3610 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00b2e0 tx=0x7fb7bc005fb0 comp rx=0 tx=0).stop 2023-07-20T02:51:59.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.158+0000 7fb7ccbff700 1 -- 172.21.15.7:0/1902595222 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7c8100760 msgr2=0x7fb7c8196450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:59.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.158+0000 7fb7ccbff700 1 --2- 172.21.15.7:0/1902595222 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7c8100760 0x7fb7c8196450 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb7b800e970 tx=0x7fb7b800ec80 comp rx=0 tx=0).stop 2023-07-20T02:51:59.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.158+0000 7fb7ccbff700 1 -- 172.21.15.7:0/1902595222 shutdown_connections 2023-07-20T02:51:59.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.158+0000 7fb7ccbff700 1 --2- 172.21.15.7:0/1902595222 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7c81022f0 0x7fb7c819da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:59.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.158+0000 7fb7ccbff700 1 --2- 172.21.15.7:0/1902595222 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fb7c8100760 0x7fb7c8196450 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:59.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.158+0000 7fb7ccbff700 1 --2- 172.21.15.7:0/1902595222 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fb7b00a1150 0x7fb7b00a3610 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:59.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.158+0000 7fb7ccbff700 1 --2- 172.21.15.7:0/1902595222 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7c8101950 0x7fb7c8196990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:59.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.158+0000 7fb7ccbff700 1 -- 172.21.15.7:0/1902595222 >> 172.21.15.7:0/1902595222 conn(0x7fb7c80fbcb0 msgr2=0x7fb7c80fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:59.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.158+0000 7fb7ccbff700 1 -- 172.21.15.7:0/1902595222 shutdown_connections 2023-07-20T02:51:59.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.158+0000 7fb7ccbff700 1 -- 172.21.15.7:0/1902595222 wait complete. 2023-07-20T02:51:59.193 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2023-07-20T02:51:59.194 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2023-07-20T02:51:59.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.391+0000 7efd11fff700 1 -- 172.21.15.7:0/81610582 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd0c101950 msgr2=0x7efd0c101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:59.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.391+0000 7efd11fff700 1 --2- 172.21.15.7:0/81610582 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd0c101950 0x7efd0c101db0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7efd000099c0 tx=0x7efd00009cd0 comp rx=0 tx=0).stop 2023-07-20T02:51:59.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.391+0000 7efd11fff700 1 -- 172.21.15.7:0/81610582 shutdown_connections 2023-07-20T02:51:59.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.391+0000 7efd11fff700 1 --2- 172.21.15.7:0/81610582 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd0c1022f0 0x7efd0c108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:59.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.391+0000 7efd11fff700 1 --2- 172.21.15.7:0/81610582 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd0c100760 0x7efd0c100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:59.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.391+0000 7efd11fff700 1 --2- 172.21.15.7:0/81610582 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd0c101950 0x7efd0c101db0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:59.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.392+0000 7efd11fff700 1 -- 172.21.15.7:0/81610582 >> 172.21.15.7:0/81610582 conn(0x7efd0c0fbcb0 msgr2=0x7efd0c0fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:51:59.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.392+0000 7efd11fff700 1 -- 172.21.15.7:0/81610582 shutdown_connections 2023-07-20T02:51:59.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.392+0000 7efd11fff700 1 -- 172.21.15.7:0/81610582 wait complete. 2023-07-20T02:51:59.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.393+0000 7efd11fff700 1 Processor -- start 2023-07-20T02:51:59.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.393+0000 7efd11fff700 1 -- start start 2023-07-20T02:51:59.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.394+0000 7efd11fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd0c100760 0x7efd0c196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:59.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.394+0000 7efd11fff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd0c101950 0x7efd0c1969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:59.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.394+0000 7efd11fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd0c1022f0 0x7efd0c19da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:59.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.394+0000 7efd11fff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0c19dff0 con 0x7efd0c1022f0 2023-07-20T02:51:59.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.394+0000 7efd11fff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0c19e130 con 0x7efd0c101950 2023-07-20T02:51:59.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.394+0000 7efd11fff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0c19e2c0 con 0x7efd0c100760 2023-07-20T02:51:59.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.394+0000 7efd0bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd0c1022f0 0x7efd0c19da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:59.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.394+0000 7efd0b7fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd0c100760 0x7efd0c196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:59.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.394+0000 7efd0bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd0c1022f0 0x7efd0c19da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46690/0 (socket says 172.21.15.7:46690) 2023-07-20T02:51:59.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.394+0000 7efd0bfff700 1 -- 172.21.15.7:0/1134214254 learned_addr learned my addr 172.21.15.7:0/1134214254 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:51:59.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.394+0000 7efd0affd700 1 --2- 172.21.15.7:0/1134214254 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd0c101950 0x7efd0c1969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:59.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.395+0000 7efd0bfff700 1 -- 172.21.15.7:0/1134214254 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd0c100760 msgr2=0x7efd0c196470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:59.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.395+0000 7efd0bfff700 1 --2- 172.21.15.7:0/1134214254 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd0c100760 0x7efd0c196470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:59.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.395+0000 7efd0bfff700 1 -- 172.21.15.7:0/1134214254 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd0c101950 msgr2=0x7efd0c1969b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:51:59.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.395+0000 7efd0bfff700 1 --2- 172.21.15.7:0/1134214254 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd0c101950 0x7efd0c1969b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:51:59.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.395+0000 7efd0bfff700 1 -- 172.21.15.7:0/1134214254 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd000095f0 con 0x7efd0c1022f0 2023-07-20T02:51:59.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.395+0000 7efd0bfff700 1 --2- 172.21.15.7:0/1134214254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd0c1022f0 0x7efd0c19da20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7efd04007ee0 tx=0x7efd0400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:59.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.395+0000 7efd08ff9700 1 -- 172.21.15.7:0/1134214254 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd04017070 con 0x7efd0c1022f0 2023-07-20T02:51:59.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.395+0000 7efd11fff700 1 -- 172.21.15.7:0/1134214254 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd0c19e5a0 con 0x7efd0c1022f0 2023-07-20T02:51:59.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.395+0000 7efd08ff9700 1 -- 172.21.15.7:0/1134214254 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efd0400f040 con 0x7efd0c1022f0 2023-07-20T02:51:59.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.395+0000 7efd08ff9700 1 -- 172.21.15.7:0/1134214254 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd04005580 con 0x7efd0c1022f0 2023-07-20T02:51:59.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.395+0000 7efd11fff700 1 -- 172.21.15.7:0/1134214254 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd0c19ea60 con 0x7efd0c1022f0 2023-07-20T02:51:59.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.396+0000 7efd11fff700 1 -- 172.21.15.7:0/1134214254 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd0c1076e0 con 0x7efd0c1022f0 2023-07-20T02:51:59.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.398+0000 7efd08ff9700 1 -- 172.21.15.7:0/1134214254 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7efd0400db20 con 0x7efd0c1022f0 2023-07-20T02:51:59.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.400+0000 7efd08ff9700 1 --2- 172.21.15.7:0/1134214254 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7efcf40a1150 0x7efcf40a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:51:59.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.400+0000 7efd08ff9700 1 -- 172.21.15.7:0/1134214254 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7efd040cc3b0 con 0x7efd0c1022f0 2023-07-20T02:51:59.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.400+0000 7efd0b7fe700 1 --2- 172.21.15.7:0/1134214254 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7efcf40a1150 0x7efcf40a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:51:59.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.401+0000 7efd0b7fe700 1 --2- 172.21.15.7:0/1134214254 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7efcf40a1150 0x7efcf40a3610 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efd0c1017b0 tx=0x7efcfc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:51:59.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.404+0000 7efd08ff9700 1 -- 172.21.15.7:0/1134214254 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7efd04095790 con 0x7efd0c1022f0 2023-07-20T02:51:59.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:59 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/1902595222' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-20T02:51:59.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:51:59 smithi007 ceph-mon[126353]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-20T02:51:59.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:59 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/1902595222' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-20T02:51:59.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:51:59 smithi188 ceph-mon[129553]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-20T02:51:59.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:59 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/1902595222' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-20T02:51:59.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:51:59 smithi158 ceph-mon[128925]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-20T02:51:59.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:51:59.664+0000 7efd11fff700 1 -- 172.21.15.7:0/1134214254 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7efd0c061420 con 0x7efd0c1022f0 2023-07-20T02:52:00.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.160+0000 7efd08ff9700 1 -- 172.21.15.7:0/1134214254 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) v1 ==== 105+0+0 (secure 0 0 0) 0x7efd04098db0 con 0x7efd0c1022f0 2023-07-20T02:52:00.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.163+0000 7efd11fff700 1 -- 172.21.15.7:0/1134214254 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7efcf40a1150 msgr2=0x7efcf40a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:52:00.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.163+0000 7efd11fff700 1 --2- 172.21.15.7:0/1134214254 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7efcf40a1150 0x7efcf40a3610 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efd0c1017b0 tx=0x7efcfc00b410 comp rx=0 tx=0).stop 2023-07-20T02:52:00.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.163+0000 7efd11fff700 1 -- 172.21.15.7:0/1134214254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd0c1022f0 msgr2=0x7efd0c19da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:52:00.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.163+0000 7efd11fff700 1 --2- 172.21.15.7:0/1134214254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd0c1022f0 0x7efd0c19da20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7efd04007ee0 tx=0x7efd0400d3b0 comp rx=0 tx=0).stop 2023-07-20T02:52:00.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.164+0000 7efd11fff700 1 -- 172.21.15.7:0/1134214254 shutdown_connections 2023-07-20T02:52:00.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.164+0000 7efd11fff700 1 --2- 172.21.15.7:0/1134214254 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7efcf40a1150 0x7efcf40a3610 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:00.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.164+0000 7efd11fff700 1 --2- 172.21.15.7:0/1134214254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd0c1022f0 0x7efd0c19da20 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:00.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.164+0000 7efd11fff700 1 --2- 172.21.15.7:0/1134214254 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd0c100760 0x7efd0c196470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:00.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.164+0000 7efd11fff700 1 --2- 172.21.15.7:0/1134214254 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7efd0c101950 0x7efd0c1969b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:00.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.164+0000 7efd11fff700 1 -- 172.21.15.7:0/1134214254 >> 172.21.15.7:0/1134214254 conn(0x7efd0c0fbcb0 msgr2=0x7efd0c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:52:00.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.164+0000 7efd11fff700 1 -- 172.21.15.7:0/1134214254 shutdown_connections 2023-07-20T02:52:00.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.164+0000 7efd11fff700 1 -- 172.21.15.7:0/1134214254 wait complete. 2023-07-20T02:52:00.201 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2023-07-20T02:52:00.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.395+0000 7f47c0bff700 1 -- 172.21.15.7:0/24019646 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47bc101950 msgr2=0x7f47bc101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:52:00.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.395+0000 7f47c0bff700 1 --2- 172.21.15.7:0/24019646 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47bc101950 0x7f47bc101db0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f47a40099c0 tx=0x7f47a4009cd0 comp rx=0 tx=0).stop 2023-07-20T02:52:00.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.395+0000 7f47c0bff700 1 -- 172.21.15.7:0/24019646 shutdown_connections 2023-07-20T02:52:00.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.395+0000 7f47c0bff700 1 --2- 172.21.15.7:0/24019646 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47bc1022f0 0x7f47bc108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:00.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.395+0000 7f47c0bff700 1 --2- 172.21.15.7:0/24019646 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47bc100760 0x7f47bc100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:00.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.395+0000 7f47c0bff700 1 --2- 172.21.15.7:0/24019646 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47bc101950 0x7f47bc101db0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:00.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.395+0000 7f47c0bff700 1 -- 172.21.15.7:0/24019646 >> 172.21.15.7:0/24019646 conn(0x7f47bc0fbcb0 msgr2=0x7f47bc0fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:52:00.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.395+0000 7f47c0bff700 1 -- 172.21.15.7:0/24019646 shutdown_connections 2023-07-20T02:52:00.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.395+0000 7f47c0bff700 1 -- 172.21.15.7:0/24019646 wait complete. 2023-07-20T02:52:00.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.396+0000 7f47c0bff700 1 Processor -- start 2023-07-20T02:52:00.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.397+0000 7f47c0bff700 1 -- start start 2023-07-20T02:52:00.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.397+0000 7f47c0bff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47bc100760 0x7f47bc196520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:52:00.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.397+0000 7f47c0bff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47bc101950 0x7f47bc196a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:52:00.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.397+0000 7f47c0bff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47bc1022f0 0x7f47bc19dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:52:00.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.397+0000 7f47c0bff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc19e0a0 con 0x7f47bc1022f0 2023-07-20T02:52:00.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.397+0000 7f47c0bff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc19e1e0 con 0x7f47bc100760 2023-07-20T02:52:00.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.397+0000 7f47c0bff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc19e370 con 0x7f47bc101950 2023-07-20T02:52:00.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.397+0000 7f47bad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47bc1022f0 0x7f47bc19dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:52:00.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.398+0000 7f47ba59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47bc100760 0x7f47bc196520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:52:00.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.398+0000 7f47b3fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47bc101950 0x7f47bc196a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:52:00.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.398+0000 7f47bad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47bc1022f0 0x7f47bc19dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46702/0 (socket says 172.21.15.7:46702) 2023-07-20T02:52:00.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.398+0000 7f47ba59c700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47bc100760 0x7f47bc196520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.7:50398/0 (socket says 172.21.15.7:50398) 2023-07-20T02:52:00.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.398+0000 7f47bad9d700 1 -- 172.21.15.7:0/3289883653 learned_addr learned my addr 172.21.15.7:0/3289883653 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:52:00.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.398+0000 7f47b3fff700 1 -- 172.21.15.7:0/3289883653 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47bc100760 msgr2=0x7f47bc196520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:52:00.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.398+0000 7f47b3fff700 1 --2- 172.21.15.7:0/3289883653 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47bc100760 0x7f47bc196520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:00.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.398+0000 7f47b3fff700 1 -- 172.21.15.7:0/3289883653 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47bc1022f0 msgr2=0x7f47bc19dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:52:00.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.398+0000 7f47b3fff700 1 --2- 172.21.15.7:0/3289883653 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47bc1022f0 0x7f47bc19dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:00.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.398+0000 7f47b3fff700 1 -- 172.21.15.7:0/3289883653 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a40095f0 con 0x7f47bc101950 2023-07-20T02:52:00.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.398+0000 7f47bad9d700 1 --2- 172.21.15.7:0/3289883653 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47bc1022f0 0x7f47bc19dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T02:52:00.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.398+0000 7f47ba59c700 1 --2- 172.21.15.7:0/3289883653 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47bc100760 0x7f47bc196520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T02:52:00.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.398+0000 7f47b3fff700 1 --2- 172.21.15.7:0/3289883653 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47bc101950 0x7f47bc196a60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f47a4009fd0 tx=0x7f47a400f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:52:00.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.398+0000 7f47b37fe700 1 -- 172.21.15.7:0/3289883653 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a401d070 con 0x7f47bc101950 2023-07-20T02:52:00.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.399+0000 7f47c0bff700 1 -- 172.21.15.7:0/3289883653 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47bc19e5f0 con 0x7f47bc101950 2023-07-20T02:52:00.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.399+0000 7f47b37fe700 1 -- 172.21.15.7:0/3289883653 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f47a4005050 con 0x7f47bc101950 2023-07-20T02:52:00.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.399+0000 7f47b37fe700 1 -- 172.21.15.7:0/3289883653 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a4004a70 con 0x7f47bc101950 2023-07-20T02:52:00.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.399+0000 7f47c0bff700 1 -- 172.21.15.7:0/3289883653 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47bc19ea70 con 0x7f47bc101950 2023-07-20T02:52:00.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.400+0000 7f47c0bff700 1 -- 172.21.15.7:0/3289883653 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47bc10bfd0 con 0x7f47bc101950 2023-07-20T02:52:00.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.402+0000 7f47b37fe700 1 -- 172.21.15.7:0/3289883653 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f47a4005450 con 0x7f47bc101950 2023-07-20T02:52:00.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.403+0000 7f47b37fe700 1 --2- 172.21.15.7:0/3289883653 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f47a80a11d0 0x7f47a80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:52:00.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.403+0000 7f47b37fe700 1 -- 172.21.15.7:0/3289883653 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f47a40cdad0 con 0x7f47bc101950 2023-07-20T02:52:00.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.404+0000 7f47ba59c700 1 --2- 172.21.15.7:0/3289883653 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f47a80a11d0 0x7f47a80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:52:00.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.404+0000 7f47ba59c700 1 --2- 172.21.15.7:0/3289883653 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f47a80a11d0 0x7f47a80a3690 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f47bc1017b0 tx=0x7f47ac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:52:00.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.405+0000 7f47b37fe700 1 -- 172.21.15.7:0/3289883653 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f47a4096f40 con 0x7f47bc101950 2023-07-20T02:52:00.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:00 smithi007 ceph-mon[126353]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-20T02:52:00.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:00 smithi007 ceph-mon[126353]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T02:52:00.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:00 smithi007 ceph-mon[126353]: pgmap v292: 129 pgs: 129 active+clean; 581 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 0 op/s 2023-07-20T02:52:00.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:00 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/1134214254' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2023-07-20T02:52:00.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:00 smithi188 ceph-mon[129553]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-20T02:52:00.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:00 smithi188 ceph-mon[129553]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T02:52:00.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:00 smithi188 ceph-mon[129553]: pgmap v292: 129 pgs: 129 active+clean; 581 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 0 op/s 2023-07-20T02:52:00.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:00 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/1134214254' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2023-07-20T02:52:00.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:00 smithi158 ceph-mon[128925]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-20T02:52:00.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:00 smithi158 ceph-mon[128925]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T02:52:00.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:00 smithi158 ceph-mon[128925]: pgmap v292: 129 pgs: 129 active+clean; 581 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 0 op/s 2023-07-20T02:52:00.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:00 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/1134214254' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2023-07-20T02:52:00.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:00.661+0000 7f47c0bff700 1 -- 172.21.15.7:0/3289883653 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f47bc107700 con 0x7f47bc101950 2023-07-20T02:52:01.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.169+0000 7f47b37fe700 1 -- 172.21.15.7:0/3289883653 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) v1 ==== 121+0+0 (secure 0 0 0) 0x7f47a409a560 con 0x7f47bc101950 2023-07-20T02:52:01.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.172+0000 7f47c0bff700 1 -- 172.21.15.7:0/3289883653 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f47a80a11d0 msgr2=0x7f47a80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:52:01.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.172+0000 7f47c0bff700 1 --2- 172.21.15.7:0/3289883653 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f47a80a11d0 0x7f47a80a3690 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f47bc1017b0 tx=0x7f47ac009260 comp rx=0 tx=0).stop 2023-07-20T02:52:01.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.172+0000 7f47c0bff700 1 -- 172.21.15.7:0/3289883653 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47bc101950 msgr2=0x7f47bc196a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:52:01.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.172+0000 7f47c0bff700 1 --2- 172.21.15.7:0/3289883653 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47bc101950 0x7f47bc196a60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f47a4009fd0 tx=0x7f47a400f690 comp rx=0 tx=0).stop 2023-07-20T02:52:01.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.172+0000 7f47c0bff700 1 -- 172.21.15.7:0/3289883653 shutdown_connections 2023-07-20T02:52:01.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.172+0000 7f47c0bff700 1 --2- 172.21.15.7:0/3289883653 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f47a80a11d0 0x7f47a80a3690 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:01.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.172+0000 7f47c0bff700 1 --2- 172.21.15.7:0/3289883653 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47bc1022f0 0x7f47bc19dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:01.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.172+0000 7f47c0bff700 1 --2- 172.21.15.7:0/3289883653 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f47bc100760 0x7f47bc196520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:01.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.172+0000 7f47c0bff700 1 --2- 172.21.15.7:0/3289883653 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47bc101950 0x7f47bc196a60 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:01.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.172+0000 7f47c0bff700 1 -- 172.21.15.7:0/3289883653 >> 172.21.15.7:0/3289883653 conn(0x7f47bc0fbcb0 msgr2=0x7f47bc0fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:52:01.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.172+0000 7f47c0bff700 1 -- 172.21.15.7:0/3289883653 shutdown_connections 2023-07-20T02:52:01.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.172+0000 7f47c0bff700 1 -- 172.21.15.7:0/3289883653 wait complete. 2023-07-20T02:52:01.209 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-07-20T02:52:01.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.405+0000 7f88741ff700 1 -- 172.21.15.7:0/117593525 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f886c100760 msgr2=0x7f886c100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:52:01.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.405+0000 7f88741ff700 1 --2- 172.21.15.7:0/117593525 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f886c100760 0x7f886c100b80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8868009920 tx=0x7f8868009c30 comp rx=0 tx=0).stop 2023-07-20T02:52:01.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.405+0000 7f88741ff700 1 -- 172.21.15.7:0/117593525 shutdown_connections 2023-07-20T02:52:01.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.405+0000 7f88741ff700 1 --2- 172.21.15.7:0/117593525 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f886c1022f0 0x7f886c108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:01.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.405+0000 7f88741ff700 1 --2- 172.21.15.7:0/117593525 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f886c100760 0x7f886c100b80 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:01.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.405+0000 7f88741ff700 1 --2- 172.21.15.7:0/117593525 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f886c101950 0x7f886c101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:01.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.405+0000 7f88741ff700 1 -- 172.21.15.7:0/117593525 >> 172.21.15.7:0/117593525 conn(0x7f886c0fbcb0 msgr2=0x7f886c0fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:52:01.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.405+0000 7f88741ff700 1 -- 172.21.15.7:0/117593525 shutdown_connections 2023-07-20T02:52:01.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.405+0000 7f88741ff700 1 -- 172.21.15.7:0/117593525 wait complete. 2023-07-20T02:52:01.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.406+0000 7f88741ff700 1 Processor -- start 2023-07-20T02:52:01.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.406+0000 7f88741ff700 1 -- start start 2023-07-20T02:52:01.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.407+0000 7f88741ff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f886c100760 0x7f886c074520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:52:01.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.407+0000 7f88741ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f886c101950 0x7f886c072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:52:01.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.407+0000 7f88741ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f886c1022f0 0x7f886c0730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:52:01.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.407+0000 7f88741ff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f886c073840 con 0x7f886c1022f0 2023-07-20T02:52:01.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.407+0000 7f88741ff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f886c0739b0 con 0x7f886c100760 2023-07-20T02:52:01.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.407+0000 7f88741ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f886c1a0590 con 0x7f886c101950 2023-07-20T02:52:01.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.407+0000 7f887279c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f886c1022f0 0x7f886c0730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:52:01.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.407+0000 7f887179a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f886c101950 0x7f886c072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:52:01.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.407+0000 7f887279c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f886c1022f0 0x7f886c0730a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46706/0 (socket says 172.21.15.7:46706) 2023-07-20T02:52:01.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.407+0000 7f887279c700 1 -- 172.21.15.7:0/1267164334 learned_addr learned my addr 172.21.15.7:0/1267164334 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:52:01.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.407+0000 7f8871f9b700 1 --2- 172.21.15.7:0/1267164334 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f886c100760 0x7f886c074520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:52:01.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.407+0000 7f887279c700 1 -- 172.21.15.7:0/1267164334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f886c101950 msgr2=0x7f886c072b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:52:01.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.407+0000 7f887279c700 1 --2- 172.21.15.7:0/1267164334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f886c101950 0x7f886c072b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:01.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.407+0000 7f887279c700 1 -- 172.21.15.7:0/1267164334 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f886c100760 msgr2=0x7f886c074520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:52:01.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.407+0000 7f887279c700 1 --2- 172.21.15.7:0/1267164334 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f886c100760 0x7f886c074520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:01.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.407+0000 7f887279c700 1 -- 172.21.15.7:0/1267164334 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8868009520 con 0x7f886c1022f0 2023-07-20T02:52:01.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.407+0000 7f887179a700 1 --2- 172.21.15.7:0/1267164334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f886c101950 0x7f886c072b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T02:52:01.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.408+0000 7f887279c700 1 --2- 172.21.15.7:0/1267164334 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f886c1022f0 0x7f886c0730a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f886400ef30 tx=0x7f886400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:52:01.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.408+0000 7f8862ffd700 1 -- 172.21.15.7:0/1267164334 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8864018070 con 0x7f886c1022f0 2023-07-20T02:52:01.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.408+0000 7f8862ffd700 1 -- 172.21.15.7:0/1267164334 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f886400f040 con 0x7f886c1022f0 2023-07-20T02:52:01.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.408+0000 7f88741ff700 1 -- 172.21.15.7:0/1267164334 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f886c1a0810 con 0x7f886c1022f0 2023-07-20T02:52:01.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.408+0000 7f8862ffd700 1 -- 172.21.15.7:0/1267164334 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886400ccf0 con 0x7f886c1022f0 2023-07-20T02:52:01.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.408+0000 7f88741ff700 1 -- 172.21.15.7:0/1267164334 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f886c1a0cd0 con 0x7f886c1022f0 2023-07-20T02:52:01.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.409+0000 7f88741ff700 1 -- 172.21.15.7:0/1267164334 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f886c107df0 con 0x7f886c1022f0 2023-07-20T02:52:01.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.411+0000 7f8862ffd700 1 -- 172.21.15.7:0/1267164334 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f8864003a00 con 0x7f886c1022f0 2023-07-20T02:52:01.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.412+0000 7f8862ffd700 1 --2- 172.21.15.7:0/1267164334 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f88580a1150 0x7f88580a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:52:01.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.412+0000 7f8862ffd700 1 -- 172.21.15.7:0/1267164334 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f88640cc7d0 con 0x7f886c1022f0 2023-07-20T02:52:01.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.413+0000 7f8871f9b700 1 --2- 172.21.15.7:0/1267164334 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f88580a1150 0x7f88580a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:52:01.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.413+0000 7f8871f9b700 1 --2- 172.21.15.7:0/1267164334 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f88580a1150 0x7f88580a3610 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f88680098f0 tx=0x7f8868019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:52:01.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.416+0000 7f8862ffd700 1 -- 172.21.15.7:0/1267164334 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8864095c40 con 0x7f886c1022f0 2023-07-20T02:52:01.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:01 smithi007 ceph-mon[126353]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-20T02:52:01.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:01 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/1134214254' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2023-07-20T02:52:01.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:01 smithi007 ceph-mon[126353]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-20T02:52:01.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:01 smithi007 ceph-mon[126353]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T02:52:01.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:01 smithi007 ceph-mon[126353]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-07-20T02:52:01.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:01 smithi007 ceph-mon[126353]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-07-20T02:52:01.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:01 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/3289883653' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-20T02:52:01.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:01 smithi007 ceph-mon[126353]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-20T02:52:01.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:01 smithi188 ceph-mon[129553]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-20T02:52:01.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:01 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/1134214254' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2023-07-20T02:52:01.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:01 smithi188 ceph-mon[129553]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-20T02:52:01.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:01 smithi188 ceph-mon[129553]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T02:52:01.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:01 smithi188 ceph-mon[129553]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-07-20T02:52:01.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:01 smithi188 ceph-mon[129553]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-07-20T02:52:01.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:01 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/3289883653' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-20T02:52:01.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:01 smithi188 ceph-mon[129553]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-20T02:52:01.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:01 smithi158 ceph-mon[128925]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-20T02:52:01.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:01 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/1134214254' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2023-07-20T02:52:01.642 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:01 smithi158 ceph-mon[128925]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-20T02:52:01.642 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:01 smithi158 ceph-mon[128925]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T02:52:01.642 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:01 smithi158 ceph-mon[128925]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-07-20T02:52:01.642 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:01 smithi158 ceph-mon[128925]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-07-20T02:52:01.642 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:01 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/3289883653' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-20T02:52:01.642 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:01 smithi158 ceph-mon[128925]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-20T02:52:01.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:01.674+0000 7f88741ff700 1 -- 172.21.15.7:0/1267164334 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f886c02c970 con 0x7f886c1022f0 2023-07-20T02:52:02.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.141+0000 7f8862ffd700 1 -- 172.21.15.7:0/1267164334 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) v1 ==== 115+0+0 (secure 0 0 0) 0x7f8864099260 con 0x7f886c1022f0 2023-07-20T02:52:02.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.144+0000 7f88741ff700 1 -- 172.21.15.7:0/1267164334 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f88580a1150 msgr2=0x7f88580a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:52:02.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.144+0000 7f88741ff700 1 --2- 172.21.15.7:0/1267164334 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f88580a1150 0x7f88580a3610 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f88680098f0 tx=0x7f8868019040 comp rx=0 tx=0).stop 2023-07-20T02:52:02.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.144+0000 7f88741ff700 1 -- 172.21.15.7:0/1267164334 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f886c1022f0 msgr2=0x7f886c0730a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:52:02.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.144+0000 7f88741ff700 1 --2- 172.21.15.7:0/1267164334 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f886c1022f0 0x7f886c0730a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f886400ef30 tx=0x7f886400c3a0 comp rx=0 tx=0).stop 2023-07-20T02:52:02.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.144+0000 7f88741ff700 1 -- 172.21.15.7:0/1267164334 shutdown_connections 2023-07-20T02:52:02.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.144+0000 7f88741ff700 1 --2- 172.21.15.7:0/1267164334 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7f88580a1150 0x7f88580a3610 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:02.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.144+0000 7f88741ff700 1 --2- 172.21.15.7:0/1267164334 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f886c1022f0 0x7f886c0730a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:02.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.144+0000 7f88741ff700 1 --2- 172.21.15.7:0/1267164334 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7f886c100760 0x7f886c074520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:02.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.144+0000 7f88741ff700 1 --2- 172.21.15.7:0/1267164334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f886c101950 0x7f886c072b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:02.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.144+0000 7f88741ff700 1 -- 172.21.15.7:0/1267164334 >> 172.21.15.7:0/1267164334 conn(0x7f886c0fbcb0 msgr2=0x7f886c104f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:52:02.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.144+0000 7f88741ff700 1 -- 172.21.15.7:0/1267164334 shutdown_connections 2023-07-20T02:52:02.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.144+0000 7f88741ff700 1 -- 172.21.15.7:0/1267164334 wait complete. 2023-07-20T02:52:02.186 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2023-07-20T02:52:02.186 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2023-07-20T02:52:02.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.382+0000 7fa5243ff700 1 -- 172.21.15.7:0/1502882836 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa51c101950 msgr2=0x7fa51c101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:52:02.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.382+0000 7fa5243ff700 1 --2- 172.21.15.7:0/1502882836 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa51c101950 0x7fa51c101db0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa50c0099c0 tx=0x7fa50c009cd0 comp rx=0 tx=0).stop 2023-07-20T02:52:02.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.382+0000 7fa5243ff700 1 -- 172.21.15.7:0/1502882836 shutdown_connections 2023-07-20T02:52:02.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.382+0000 7fa5243ff700 1 --2- 172.21.15.7:0/1502882836 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa51c1022f0 0x7fa51c108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:02.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.382+0000 7fa5243ff700 1 --2- 172.21.15.7:0/1502882836 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa51c100760 0x7fa51c100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:02.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.382+0000 7fa5243ff700 1 --2- 172.21.15.7:0/1502882836 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa51c101950 0x7fa51c101db0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:02.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.382+0000 7fa5243ff700 1 -- 172.21.15.7:0/1502882836 >> 172.21.15.7:0/1502882836 conn(0x7fa51c0fbcb0 msgr2=0x7fa51c0fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:52:02.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.382+0000 7fa5243ff700 1 -- 172.21.15.7:0/1502882836 shutdown_connections 2023-07-20T02:52:02.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.382+0000 7fa5243ff700 1 -- 172.21.15.7:0/1502882836 wait complete. 2023-07-20T02:52:02.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.383+0000 7fa5243ff700 1 Processor -- start 2023-07-20T02:52:02.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.383+0000 7fa5243ff700 1 -- start start 2023-07-20T02:52:02.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.383+0000 7fa5243ff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa51c100760 0x7fa51c074520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:52:02.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.383+0000 7fa5243ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa51c101950 0x7fa51c072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:52:02.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.383+0000 7fa5243ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa51c1022f0 0x7fa51c0730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:52:02.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.383+0000 7fa5243ff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51c19a330 con 0x7fa51c101950 2023-07-20T02:52:02.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.383+0000 7fa5243ff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51c0735e0 con 0x7fa51c100760 2023-07-20T02:52:02.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.383+0000 7fa5243ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51c0737a0 con 0x7fa51c1022f0 2023-07-20T02:52:02.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.384+0000 7fa52219b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa51c100760 0x7fa51c074520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:52:02.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.384+0000 7fa52199a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa51c101950 0x7fa51c072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:52:02.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.384+0000 7fa52219b700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa51c100760 0x7fa51c074520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.158:3300/0 says I am v2:172.21.15.7:34218/0 (socket says 172.21.15.7:34218) 2023-07-20T02:52:02.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.384+0000 7fa52219b700 1 -- 172.21.15.7:0/12787499 learned_addr learned my addr 172.21.15.7:0/12787499 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:52:02.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.384+0000 7fa52299c700 1 --2- 172.21.15.7:0/12787499 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa51c1022f0 0x7fa51c0730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:52:02.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.384+0000 7fa52219b700 1 -- 172.21.15.7:0/12787499 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa51c1022f0 msgr2=0x7fa51c0730a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:52:02.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.384+0000 7fa52219b700 1 --2- 172.21.15.7:0/12787499 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa51c1022f0 0x7fa51c0730a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:02.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.384+0000 7fa52219b700 1 -- 172.21.15.7:0/12787499 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa51c101950 msgr2=0x7fa51c072b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:52:02.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.384+0000 7fa52219b700 1 --2- 172.21.15.7:0/12787499 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa51c101950 0x7fa51c072b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:02.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.384+0000 7fa52219b700 1 -- 172.21.15.7:0/12787499 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa50c0095f0 con 0x7fa51c100760 2023-07-20T02:52:02.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.384+0000 7fa52219b700 1 --2- 172.21.15.7:0/12787499 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa51c100760 0x7fa51c074520 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa51800e970 tx=0x7fa51800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:52:02.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.385+0000 7fa5137fe700 1 -- 172.21.15.7:0/12787499 <== mon.2 v2:172.21.15.158:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa518009790 con 0x7fa51c100760 2023-07-20T02:52:02.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.385+0000 7fa5137fe700 1 -- 172.21.15.7:0/12787499 <== mon.2 v2:172.21.15.158:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa51800c7e0 con 0x7fa51c100760 2023-07-20T02:52:02.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.385+0000 7fa5243ff700 1 -- 172.21.15.7:0/12787499 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa51c1a4ce0 con 0x7fa51c100760 2023-07-20T02:52:02.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.385+0000 7fa5137fe700 1 -- 172.21.15.7:0/12787499 <== mon.2 v2:172.21.15.158:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa518003c70 con 0x7fa51c100760 2023-07-20T02:52:02.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.385+0000 7fa5243ff700 1 -- 172.21.15.7:0/12787499 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa51c1a5100 con 0x7fa51c100760 2023-07-20T02:52:02.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.386+0000 7fa5243ff700 1 -- 172.21.15.7:0/12787499 --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa51c10bfd0 con 0x7fa51c100760 2023-07-20T02:52:02.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.388+0000 7fa5137fe700 1 -- 172.21.15.7:0/12787499 <== mon.2 v2:172.21.15.158:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fa51800cc50 con 0x7fa51c100760 2023-07-20T02:52:02.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.389+0000 7fa5137fe700 1 --2- 172.21.15.7:0/12787499 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fa5080a1150 0x7fa5080a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:52:02.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.389+0000 7fa5137fe700 1 -- 172.21.15.7:0/12787499 <== mon.2 v2:172.21.15.158:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa518014070 con 0x7fa51c100760 2023-07-20T02:52:02.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.389+0000 7fa52199a700 1 --2- 172.21.15.7:0/12787499 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fa5080a1150 0x7fa5080a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:52:02.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.390+0000 7fa52199a700 1 --2- 172.21.15.7:0/12787499 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fa5080a1150 0x7fa5080a3610 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa50c009fd0 tx=0x7fa50c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:52:02.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.391+0000 7fa5137fe700 1 -- 172.21.15.7:0/12787499 <== mon.2 v2:172.21.15.158:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa5180967f0 con 0x7fa51c100760 2023-07-20T02:52:02.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:02 smithi007 ceph-mon[126353]: mds.? [v2:172.21.15.188:6838/450074084,v1:172.21.15.188:6839/450074084] up:active 2023-07-20T02:52:02.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:02 smithi007 ceph-mon[126353]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-20T02:52:02.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:02 smithi007 ceph-mon[126353]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-20T02:52:02.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:02 smithi007 ceph-mon[126353]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-07-20T02:52:02.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:02 smithi007 ceph-mon[126353]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2023-07-20T02:52:02.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:02 smithi007 ceph-mon[126353]: daemon mds.c is now active in filesystem cephfs as rank 2 2023-07-20T02:52:02.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:02 smithi007 ceph-mon[126353]: pgmap v293: 129 pgs: 129 active+clean; 582 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2023-07-20T02:52:02.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:02 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/1267164334' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-20T02:52:02.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:02 smithi007 ceph-mon[126353]: mds.? [v2:172.21.15.188:6836/775436728,v1:172.21.15.188:6837/775436728] up:active 2023-07-20T02:52:02.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:02 smithi007 ceph-mon[126353]: from='client.? 172.21.15.7:0/1267164334' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-20T02:52:02.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:02 smithi007 ceph-mon[126353]: daemon mds.j assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2023-07-20T02:52:02.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:02 smithi007 ceph-mon[126353]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-07-20T02:52:02.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:02 smithi007 ceph-mon[126353]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=j=up:creating} 3 up:standby-replay 5 up:standby 2023-07-20T02:52:02.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:02 smithi007 ceph-mon[126353]: daemon mds.j is now active in filesystem cephfs as rank 3 2023-07-20T02:52:02.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:02 smithi188 ceph-mon[129553]: mds.? [v2:172.21.15.188:6838/450074084,v1:172.21.15.188:6839/450074084] up:active 2023-07-20T02:52:02.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:02 smithi188 ceph-mon[129553]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-20T02:52:02.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:02 smithi188 ceph-mon[129553]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-20T02:52:02.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:02 smithi188 ceph-mon[129553]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-07-20T02:52:02.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:02 smithi188 ceph-mon[129553]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2023-07-20T02:52:02.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:02 smithi188 ceph-mon[129553]: daemon mds.c is now active in filesystem cephfs as rank 2 2023-07-20T02:52:02.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:02 smithi188 ceph-mon[129553]: pgmap v293: 129 pgs: 129 active+clean; 582 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2023-07-20T02:52:02.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:02 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/1267164334' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-20T02:52:02.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:02 smithi188 ceph-mon[129553]: mds.? [v2:172.21.15.188:6836/775436728,v1:172.21.15.188:6837/775436728] up:active 2023-07-20T02:52:02.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:02 smithi188 ceph-mon[129553]: from='client.? 172.21.15.7:0/1267164334' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-20T02:52:02.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:02 smithi188 ceph-mon[129553]: daemon mds.j assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2023-07-20T02:52:02.466 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:02 smithi188 ceph-mon[129553]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-07-20T02:52:02.466 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:02 smithi188 ceph-mon[129553]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=j=up:creating} 3 up:standby-replay 5 up:standby 2023-07-20T02:52:02.466 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:02 smithi188 ceph-mon[129553]: daemon mds.j is now active in filesystem cephfs as rank 3 2023-07-20T02:52:02.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:02 smithi158 ceph-mon[128925]: mds.? [v2:172.21.15.188:6838/450074084,v1:172.21.15.188:6839/450074084] up:active 2023-07-20T02:52:02.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:02 smithi158 ceph-mon[128925]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-20T02:52:02.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:02 smithi158 ceph-mon[128925]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-20T02:52:02.642 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:02 smithi158 ceph-mon[128925]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-07-20T02:52:02.642 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:02 smithi158 ceph-mon[128925]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2023-07-20T02:52:02.642 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:02 smithi158 ceph-mon[128925]: daemon mds.c is now active in filesystem cephfs as rank 2 2023-07-20T02:52:02.642 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:02 smithi158 ceph-mon[128925]: pgmap v293: 129 pgs: 129 active+clean; 582 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2023-07-20T02:52:02.642 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:02 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/1267164334' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-20T02:52:02.642 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:02 smithi158 ceph-mon[128925]: mds.? [v2:172.21.15.188:6836/775436728,v1:172.21.15.188:6837/775436728] up:active 2023-07-20T02:52:02.642 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:02 smithi158 ceph-mon[128925]: from='client.? 172.21.15.7:0/1267164334' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-20T02:52:02.643 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:02 smithi158 ceph-mon[128925]: daemon mds.j assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2023-07-20T02:52:02.643 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:02 smithi158 ceph-mon[128925]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-07-20T02:52:02.643 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:02 smithi158 ceph-mon[128925]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=j=up:creating} 3 up:standby-replay 5 up:standby 2023-07-20T02:52:02.643 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:02 smithi158 ceph-mon[128925]: daemon mds.j is now active in filesystem cephfs as rank 3 2023-07-20T02:52:02.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:02.630+0000 7fa5243ff700 1 -- 172.21.15.7:0/12787499 --> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7fa51c060bb0 con 0x7fa5080a1150 2023-07-20T02:52:02.948 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:52:02 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:52:02.636+0000 7f60b7a7a700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T02:52:02.949 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:52:02 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:52:02.636+0000 7f60b7a7a700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T02:52:02.949 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:52:02 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:52:02.636+0000 7f60b7a7a700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T02:52:02.949 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:52:02 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:52:02.636+0000 7f60b7a7a700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T02:52:02.949 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:52:02 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:52:02.636+0000 7f60b7a7a700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T02:52:03.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.168+0000 7fa5137fe700 1 -- 172.21.15.7:0/12787499 <== mgr.14152 v2:172.21.15.7:6800/1354542137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa51c060bb0 con 0x7fa5080a1150 2023-07-20T02:52:03.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.171+0000 7fa5243ff700 1 -- 172.21.15.7:0/12787499 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fa5080a1150 msgr2=0x7fa5080a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:52:03.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.171+0000 7fa5243ff700 1 --2- 172.21.15.7:0/12787499 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fa5080a1150 0x7fa5080a3610 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa50c009fd0 tx=0x7fa50c019040 comp rx=0 tx=0).stop 2023-07-20T02:52:03.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.171+0000 7fa5243ff700 1 -- 172.21.15.7:0/12787499 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa51c100760 msgr2=0x7fa51c074520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:52:03.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.171+0000 7fa5243ff700 1 --2- 172.21.15.7:0/12787499 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa51c100760 0x7fa51c074520 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa51800e970 tx=0x7fa51800ec80 comp rx=0 tx=0).stop 2023-07-20T02:52:03.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.171+0000 7fa5243ff700 1 -- 172.21.15.7:0/12787499 shutdown_connections 2023-07-20T02:52:03.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.171+0000 7fa5243ff700 1 --2- 172.21.15.7:0/12787499 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa51c1022f0 0x7fa51c0730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:03.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.171+0000 7fa5243ff700 1 --2- 172.21.15.7:0/12787499 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa51c100760 0x7fa51c074520 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:03.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.171+0000 7fa5243ff700 1 --2- 172.21.15.7:0/12787499 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fa5080a1150 0x7fa5080a3610 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:03.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.171+0000 7fa5243ff700 1 --2- 172.21.15.7:0/12787499 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa51c101950 0x7fa51c072b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:03.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.171+0000 7fa5243ff700 1 -- 172.21.15.7:0/12787499 >> 172.21.15.7:0/12787499 conn(0x7fa51c0fbcb0 msgr2=0x7fa51c0fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:52:03.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.171+0000 7fa5243ff700 1 -- 172.21.15.7:0/12787499 shutdown_connections 2023-07-20T02:52:03.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.171+0000 7fa5243ff700 1 -- 172.21.15.7:0/12787499 wait complete. 2023-07-20T02:52:03.206 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs --group_name=qa sv_0 '' 2023-07-20T02:52:03.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.402+0000 7fbcdebff700 1 -- 172.21.15.7:0/1094455040 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbcd8101950 msgr2=0x7fbcd8101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:52:03.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.402+0000 7fbcdebff700 1 --2- 172.21.15.7:0/1094455040 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbcd8101950 0x7fbcd8101db0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbcc40099c0 tx=0x7fbcc4009cd0 comp rx=0 tx=0).stop 2023-07-20T02:52:03.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.402+0000 7fbcdebff700 1 -- 172.21.15.7:0/1094455040 shutdown_connections 2023-07-20T02:52:03.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.402+0000 7fbcdebff700 1 --2- 172.21.15.7:0/1094455040 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbcd81022f0 0x7fbcd8108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:03.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.402+0000 7fbcdebff700 1 --2- 172.21.15.7:0/1094455040 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbcd8100760 0x7fbcd8100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:03.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.402+0000 7fbcdebff700 1 --2- 172.21.15.7:0/1094455040 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbcd8101950 0x7fbcd8101db0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:03.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.402+0000 7fbcdebff700 1 -- 172.21.15.7:0/1094455040 >> 172.21.15.7:0/1094455040 conn(0x7fbcd80fbcb0 msgr2=0x7fbcd80fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:52:03.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.402+0000 7fbcdebff700 1 -- 172.21.15.7:0/1094455040 shutdown_connections 2023-07-20T02:52:03.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.402+0000 7fbcdebff700 1 -- 172.21.15.7:0/1094455040 wait complete. 2023-07-20T02:52:03.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.403+0000 7fbcdebff700 1 Processor -- start 2023-07-20T02:52:03.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.403+0000 7fbcdebff700 1 -- start start 2023-07-20T02:52:03.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.403+0000 7fbcdebff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbcd8100760 0x7fbcd81986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:52:03.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.403+0000 7fbcdebff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbcd8101950 0x7fbcd8198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:52:03.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.404+0000 7fbcdebff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbcd81022f0 0x7fbcd819fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:52:03.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.404+0000 7fbcdebff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd81a0220 con 0x7fbcd81022f0 2023-07-20T02:52:03.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.404+0000 7fbcdebff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd81a0360 con 0x7fbcd8100760 2023-07-20T02:52:03.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.404+0000 7fbcdebff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd81a04f0 con 0x7fbcd8101950 2023-07-20T02:52:03.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.404+0000 7fbcdd19c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbcd81022f0 0x7fbcd819fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:52:03.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.404+0000 7fbcdd19c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbcd81022f0 0x7fbcd819fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48346/0 (socket says 172.21.15.7:48346) 2023-07-20T02:52:03.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.404+0000 7fbcdd19c700 1 -- 172.21.15.7:0/2945429862 learned_addr learned my addr 172.21.15.7:0/2945429862 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:52:03.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.404+0000 7fbcdc99b700 1 --2- 172.21.15.7:0/2945429862 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbcd8100760 0x7fbcd81986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:52:03.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.404+0000 7fbcd7fff700 1 --2- 172.21.15.7:0/2945429862 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbcd8101950 0x7fbcd8198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:52:03.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.404+0000 7fbcdd19c700 1 -- 172.21.15.7:0/2945429862 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbcd8101950 msgr2=0x7fbcd8198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:52:03.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.404+0000 7fbcdd19c700 1 --2- 172.21.15.7:0/2945429862 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbcd8101950 0x7fbcd8198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:03.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.404+0000 7fbcdd19c700 1 -- 172.21.15.7:0/2945429862 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbcd8100760 msgr2=0x7fbcd81986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:52:03.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.404+0000 7fbcdd19c700 1 --2- 172.21.15.7:0/2945429862 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbcd8100760 0x7fbcd81986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:03.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.404+0000 7fbcdd19c700 1 -- 172.21.15.7:0/2945429862 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc40095f0 con 0x7fbcd81022f0 2023-07-20T02:52:03.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.404+0000 7fbcdd19c700 1 --2- 172.21.15.7:0/2945429862 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbcd81022f0 0x7fbcd819fc50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbcd000ef30 tx=0x7fbcd000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:52:03.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.405+0000 7fbcd5ffb700 1 -- 172.21.15.7:0/2945429862 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd0018070 con 0x7fbcd81022f0 2023-07-20T02:52:03.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.405+0000 7fbcdebff700 1 -- 172.21.15.7:0/2945429862 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd81a0770 con 0x7fbcd81022f0 2023-07-20T02:52:03.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.405+0000 7fbcd5ffb700 1 -- 172.21.15.7:0/2945429862 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbcd000f040 con 0x7fbcd81022f0 2023-07-20T02:52:03.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.405+0000 7fbcd5ffb700 1 -- 172.21.15.7:0/2945429862 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd000cd00 con 0x7fbcd81022f0 2023-07-20T02:52:03.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.405+0000 7fbcdebff700 1 -- 172.21.15.7:0/2945429862 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcd81a0c30 con 0x7fbcd81022f0 2023-07-20T02:52:03.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.406+0000 7fbcdebff700 1 -- 172.21.15.7:0/2945429862 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcd810bfd0 con 0x7fbcd81022f0 2023-07-20T02:52:03.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.408+0000 7fbcd5ffb700 1 -- 172.21.15.7:0/2945429862 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fbcd0003a00 con 0x7fbcd81022f0 2023-07-20T02:52:03.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.409+0000 7fbcd5ffb700 1 --2- 172.21.15.7:0/2945429862 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fbcc80a1150 0x7fbcc80a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:52:03.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.410+0000 7fbcd5ffb700 1 -- 172.21.15.7:0/2945429862 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fbcd00cc7d0 con 0x7fbcd81022f0 2023-07-20T02:52:03.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.410+0000 7fbcdc99b700 1 --2- 172.21.15.7:0/2945429862 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fbcc80a1150 0x7fbcc80a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:52:03.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.410+0000 7fbcdc99b700 1 --2- 172.21.15.7:0/2945429862 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fbcc80a1150 0x7fbcc80a3610 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbcd81017b0 tx=0x7fbccc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:52:03.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.414+0000 7fbcd5ffb700 1 -- 172.21.15.7:0/2945429862 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fbcd0095c40 con 0x7fbcd81022f0 2023-07-20T02:52:03.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:03 smithi007 ceph-mon[126353]: from='client.24812 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:52:03.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:03 smithi007 ceph-mon[126353]: mds.? [v2:172.21.15.7:6834/4064058311,v1:172.21.15.7:6835/4064058311] up:active 2023-07-20T02:52:03.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:03 smithi007 ceph-mon[126353]: daemon mds.k assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2023-07-20T02:52:03.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:03 smithi007 ceph-mon[126353]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-20T02:52:03.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:03 smithi007 ceph-mon[126353]: Cluster is now healthy 2023-07-20T02:52:03.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:03 smithi007 ceph-mon[126353]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=j=up:active} 3 up:standby-replay 5 up:standby 2023-07-20T02:52:03.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:03 smithi007 ceph-mon[126353]: fsmap cephfs:5 3 up:standby-replay 1 up:creating 4 up:standby 4 up:active 2023-07-20T02:52:03.451 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:03 smithi007 ceph-mon[126353]: daemon mds.k is now active in filesystem cephfs as rank 4 2023-07-20T02:52:03.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:03 smithi188 ceph-mon[129553]: from='client.24812 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:52:03.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:03 smithi188 ceph-mon[129553]: mds.? [v2:172.21.15.7:6834/4064058311,v1:172.21.15.7:6835/4064058311] up:active 2023-07-20T02:52:03.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:03 smithi188 ceph-mon[129553]: daemon mds.k assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2023-07-20T02:52:03.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:03 smithi188 ceph-mon[129553]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-20T02:52:03.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:03 smithi188 ceph-mon[129553]: Cluster is now healthy 2023-07-20T02:52:03.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:03 smithi188 ceph-mon[129553]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=j=up:active} 3 up:standby-replay 5 up:standby 2023-07-20T02:52:03.466 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:03 smithi188 ceph-mon[129553]: fsmap cephfs:5 3 up:standby-replay 1 up:creating 4 up:standby 4 up:active 2023-07-20T02:52:03.466 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:03 smithi188 ceph-mon[129553]: daemon mds.k is now active in filesystem cephfs as rank 4 2023-07-20T02:52:03.641 INFO:journalctl@ceph.mds.b.smithi158.stdout:Jul 20 02:52:03 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-b[154225]: 2023-07-20T02:52:03.137+0000 7f091b11f700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-07-20T02:52:03.641 INFO:journalctl@ceph.mds.b.smithi158.stdout:Jul 20 02:52:03 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-b[154225]: 2023-07-20T02:52:03.137+0000 7f091b11f700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-07-20T02:52:03.642 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:03 smithi158 ceph-mon[128925]: from='client.24812 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:52:03.642 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:03 smithi158 ceph-mon[128925]: mds.? [v2:172.21.15.7:6834/4064058311,v1:172.21.15.7:6835/4064058311] up:active 2023-07-20T02:52:03.642 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:03 smithi158 ceph-mon[128925]: daemon mds.k assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2023-07-20T02:52:03.642 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:03 smithi158 ceph-mon[128925]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-20T02:52:03.642 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:03 smithi158 ceph-mon[128925]: Cluster is now healthy 2023-07-20T02:52:03.642 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:03 smithi158 ceph-mon[128925]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=j=up:active} 3 up:standby-replay 5 up:standby 2023-07-20T02:52:03.642 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:03 smithi158 ceph-mon[128925]: fsmap cephfs:5 3 up:standby-replay 1 up:creating 4 up:standby 4 up:active 2023-07-20T02:52:03.643 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:03 smithi158 ceph-mon[128925]: daemon mds.k is now active in filesystem cephfs as rank 4 2023-07-20T02:52:03.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.656+0000 7fbcdebff700 1 -- 172.21.15.7:0/2945429862 --> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "group_name": "", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7fbcd8108340 con 0x7fbcc80a1150 2023-07-20T02:52:03.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.687+0000 7fbcd5ffb700 1 -- 172.21.15.7:0/2945429862 <== mgr.14152 v2:172.21.15.7:6800/1354542137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbcd8108340 con 0x7fbcc80a1150 2023-07-20T02:52:03.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.689+0000 7fbcdebff700 1 -- 172.21.15.7:0/2945429862 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fbcc80a1150 msgr2=0x7fbcc80a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:52:03.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.689+0000 7fbcdebff700 1 --2- 172.21.15.7:0/2945429862 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fbcc80a1150 0x7fbcc80a3610 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbcd81017b0 tx=0x7fbccc009260 comp rx=0 tx=0).stop 2023-07-20T02:52:03.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.689+0000 7fbcdebff700 1 -- 172.21.15.7:0/2945429862 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbcd81022f0 msgr2=0x7fbcd819fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:52:03.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.689+0000 7fbcdebff700 1 --2- 172.21.15.7:0/2945429862 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbcd81022f0 0x7fbcd819fc50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbcd000ef30 tx=0x7fbcd000c3a0 comp rx=0 tx=0).stop 2023-07-20T02:52:03.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.689+0000 7fbcdebff700 1 -- 172.21.15.7:0/2945429862 shutdown_connections 2023-07-20T02:52:03.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.689+0000 7fbcdebff700 1 --2- 172.21.15.7:0/2945429862 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fbcc80a1150 0x7fbcc80a3610 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:03.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.689+0000 7fbcdebff700 1 --2- 172.21.15.7:0/2945429862 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbcd81022f0 0x7fbcd819fc50 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:03.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.689+0000 7fbcdebff700 1 --2- 172.21.15.7:0/2945429862 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fbcd8100760 0x7fbcd81986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:03.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.689+0000 7fbcdebff700 1 --2- 172.21.15.7:0/2945429862 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbcd8101950 0x7fbcd8198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:03.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.689+0000 7fbcdebff700 1 -- 172.21.15.7:0/2945429862 >> 172.21.15.7:0/2945429862 conn(0x7fbcd80fbcb0 msgr2=0x7fbcd80fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:52:03.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.689+0000 7fbcdebff700 1 -- 172.21.15.7:0/2945429862 shutdown_connections 2023-07-20T02:52:03.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.690+0000 7fbcdebff700 1 -- 172.21.15.7:0/2945429862 wait complete. 2023-07-20T02:52:03.722 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name=qa sv_0 2023-07-20T02:52:03.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.917+0000 7fa7761ff700 1 -- 172.21.15.7:0/1790590786 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa770073170 msgr2=0x7fa7700735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:52:03.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.917+0000 7fa7761ff700 1 --2- 172.21.15.7:0/1790590786 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa770073170 0x7fa7700735f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa764009990 tx=0x7fa764009ca0 comp rx=0 tx=0).stop 2023-07-20T02:52:03.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.917+0000 7fa7761ff700 1 -- 172.21.15.7:0/1790590786 shutdown_connections 2023-07-20T02:52:03.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.917+0000 7fa7761ff700 1 --2- 172.21.15.7:0/1790590786 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa770108630 0x7fa77010aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:03.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.917+0000 7fa7761ff700 1 --2- 172.21.15.7:0/1790590786 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa770074750 0x7fa770072ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:03.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.917+0000 7fa7761ff700 1 --2- 172.21.15.7:0/1790590786 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa770073170 0x7fa7700735f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:03.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.917+0000 7fa7761ff700 1 -- 172.21.15.7:0/1790590786 >> 172.21.15.7:0/1790590786 conn(0x7fa7700fbcb0 msgr2=0x7fa7700fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T02:52:03.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.917+0000 7fa7761ff700 1 -- 172.21.15.7:0/1790590786 shutdown_connections 2023-07-20T02:52:03.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.917+0000 7fa7761ff700 1 -- 172.21.15.7:0/1790590786 wait complete. 2023-07-20T02:52:03.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.918+0000 7fa7761ff700 1 Processor -- start 2023-07-20T02:52:03.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.918+0000 7fa7761ff700 1 -- start start 2023-07-20T02:52:03.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.918+0000 7fa7761ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa770073170 0x7fa770198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:52:03.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.918+0000 7fa7761ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa770074750 0x7fa770198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:52:03.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.918+0000 7fa7761ff700 1 --2- >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa770108630 0x7fa77019fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:52:03.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.918+0000 7fa7761ff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7701a01f0 con 0x7fa770074750 2023-07-20T02:52:03.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.918+0000 7fa7761ff700 1 -- --> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7701a0330 con 0x7fa770108630 2023-07-20T02:52:03.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.918+0000 7fa7761ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7701a04c0 con 0x7fa770073170 2023-07-20T02:52:03.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.919+0000 7fa76effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa770074750 0x7fa770198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:52:03.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.919+0000 7fa76effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa770074750 0x7fa770198bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48358/0 (socket says 172.21.15.7:48358) 2023-07-20T02:52:03.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.919+0000 7fa76effd700 1 -- 172.21.15.7:0/2713940760 learned_addr learned my addr 172.21.15.7:0/2713940760 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-07-20T02:52:03.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.919+0000 7fa76ffff700 1 --2- 172.21.15.7:0/2713940760 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa770108630 0x7fa77019fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:52:03.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.919+0000 7fa76f7fe700 1 --2- 172.21.15.7:0/2713940760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa770073170 0x7fa770198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:52:03.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.919+0000 7fa76effd700 1 -- 172.21.15.7:0/2713940760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa770073170 msgr2=0x7fa770198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:52:03.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.919+0000 7fa76effd700 1 --2- 172.21.15.7:0/2713940760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa770073170 0x7fa770198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:03.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.919+0000 7fa76effd700 1 -- 172.21.15.7:0/2713940760 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa770108630 msgr2=0x7fa77019fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T02:52:03.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.919+0000 7fa76effd700 1 --2- 172.21.15.7:0/2713940760 >> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] conn(0x7fa770108630 0x7fa77019fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T02:52:03.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.919+0000 7fa76effd700 1 -- 172.21.15.7:0/2713940760 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7640095f0 con 0x7fa770074750 2023-07-20T02:52:03.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.919+0000 7fa76effd700 1 --2- 172.21.15.7:0/2713940760 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa770074750 0x7fa770198bb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa76400b680 tx=0x7fa764005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:52:03.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.920+0000 7fa76cff9700 1 -- 172.21.15.7:0/2713940760 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76401d070 con 0x7fa770074750 2023-07-20T02:52:03.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.920+0000 7fa7761ff700 1 -- 172.21.15.7:0/2713940760 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7701a0740 con 0x7fa770074750 2023-07-20T02:52:03.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.920+0000 7fa76cff9700 1 -- 172.21.15.7:0/2713940760 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa764022470 con 0x7fa770074750 2023-07-20T02:52:03.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.920+0000 7fa76cff9700 1 -- 172.21.15.7:0/2713940760 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa764017650 con 0x7fa770074750 2023-07-20T02:52:03.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.920+0000 7fa7761ff700 1 -- 172.21.15.7:0/2713940760 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7701a0bc0 con 0x7fa770074750 2023-07-20T02:52:03.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.921+0000 7fa7761ff700 1 -- 172.21.15.7:0/2713940760 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa770106d30 con 0x7fa770074750 2023-07-20T02:52:03.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.923+0000 7fa76cff9700 1 -- 172.21.15.7:0/2713940760 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fa7640225e0 con 0x7fa770074750 2023-07-20T02:52:03.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.924+0000 7fa76cff9700 1 --2- 172.21.15.7:0/2713940760 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fa7580a1150 0x7fa7580a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T02:52:03.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.924+0000 7fa76cff9700 1 -- 172.21.15.7:0/2713940760 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa7640ce100 con 0x7fa770074750 2023-07-20T02:52:03.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.924+0000 7fa76f7fe700 1 --2- 172.21.15.7:0/2713940760 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fa7580a1150 0x7fa7580a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T02:52:03.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.925+0000 7fa76f7fe700 1 --2- 172.21.15.7:0/2713940760 >> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] conn(0x7fa7580a1150 0x7fa7580a3610 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa760009c60 tx=0x7fa760009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T02:52:03.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:03.928+0000 7fa76cff9700 1 -- 172.21.15.7:0/2713940760 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa764097570 con 0x7fa770074750 2023-07-20T02:52:04.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:04.177+0000 7fa7761ff700 1 -- 172.21.15.7:0/2713940760 --> [v2:172.21.15.7:6800/1354542137,v1:172.21.15.7:6801/1354542137] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7fa770060bb0 con 0x7fa7580a1150 2023-07-20T02:52:04.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-07-20T02:52:04.180+0000 7fa76cff9700 1 -- 172.21.15.7:0/2713940760 <== mgr.14152 v2:172.21.15.7:6800/1354542137 1 ==== mgr_command_reply(tid 0: -2 subvolume 'sv_0' does not exist) v1 ==== 39+0+0 (secure 0 0 0) 0x7fa770060bb0 con 0x7fa7580a1150 2023-07-20T02:52:04.196 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENOENT: subvolume 'sv_0' does not exist 2023-07-20T02:52:04.210 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-07-20T02:52:04.211 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephadm.py", line 980, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/filesystem.py", line 535, in __init__ self.create() File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/filesystem.py", line 773, in create subvol_path = self.get_ceph_cmd_stdout( File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/cephfs_test_case.py", line 73, in get_ceph_cmd_stdout return self.run_ceph_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/cephfs_test_case.py", line 58, in run_ceph_cmd return self.mon_manager.run_cluster_cmd(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 2: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name=qa sv_0' 2023-07-20T02:52:04.211 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T02:52:04.282 DEBUG:teuthology.orchestra.run.smithi158:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T02:52:04.317 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T02:52:04.355 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-07-20T02:52:04.355 DEBUG:teuthology.orchestra.run.smithi007:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-20T02:52:04.399 DEBUG:teuthology.orchestra.run.smithi158:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-20T02:52:04.417 DEBUG:teuthology.orchestra.run.smithi188:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-20T02:52:04.437 INFO:tasks.cephadm:Stopping all daemons... 2023-07-20T02:52:04.437 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2023-07-20T02:52:04.438 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mon.a 2023-07-20T02:52:04.448 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:52:04 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:52:04.180+0000 7f60b7a7a700 -1 mgr.server reply reply (2) No such file or directory subvolume 'sv_0' does not exist 2023-07-20T02:52:04.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:04 smithi007 ceph-mon[126353]: pgmap v294: 129 pgs: 129 active+clean; 583 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 722 B/s rd, 1.4 KiB/s wr, 7 op/s 2023-07-20T02:52:04.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:04 smithi007 ceph-mon[126353]: from='client.14829 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "group_name": "", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:52:04.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:04 smithi007 ceph-mon[126353]: mds.? [v2:172.21.15.158:6832/401861265,v1:172.21.15.158:6833/401861265] up:active 2023-07-20T02:52:04.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:04 smithi007 ceph-mon[126353]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2023-07-20T02:52:04.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:04 smithi007 ceph-mon[126353]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2023-07-20T02:52:04.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:04 smithi188 ceph-mon[129553]: pgmap v294: 129 pgs: 129 active+clean; 583 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 722 B/s rd, 1.4 KiB/s wr, 7 op/s 2023-07-20T02:52:04.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:04 smithi188 ceph-mon[129553]: from='client.14829 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "group_name": "", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:52:04.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:04 smithi188 ceph-mon[129553]: mds.? [v2:172.21.15.158:6832/401861265,v1:172.21.15.158:6833/401861265] up:active 2023-07-20T02:52:04.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:04 smithi188 ceph-mon[129553]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2023-07-20T02:52:04.464 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:04 smithi188 ceph-mon[129553]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2023-07-20T02:52:04.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:04 smithi158 ceph-mon[128925]: pgmap v294: 129 pgs: 129 active+clean; 583 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 722 B/s rd, 1.4 KiB/s wr, 7 op/s 2023-07-20T02:52:04.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:04 smithi158 ceph-mon[128925]: from='client.14829 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "group_name": "", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T02:52:04.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:04 smithi158 ceph-mon[128925]: mds.? [v2:172.21.15.158:6832/401861265,v1:172.21.15.158:6833/401861265] up:active 2023-07-20T02:52:04.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:04 smithi158 ceph-mon[128925]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2023-07-20T02:52:04.642 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:04 smithi158 ceph-mon[128925]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2023-07-20T02:52:04.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:04 smithi007 systemd[1]: Stopping Ceph mon.a for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:52:05.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:04 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mon-a[126329]: 2023-07-20T02:52:04.935+0000 7f1bb476f700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:52:05.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:04 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mon-a[126329]: 2023-07-20T02:52:04.935+0000 7f1bb476f700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2023-07-20T02:52:05.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Jul 20 02:52:05 smithi007 bash[165255]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mon-a 2023-07-20T02:52:05.846 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mon.a.service' 2023-07-20T02:52:05.995 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:52:05.996 INFO:tasks.cephadm.mon.a:Stopped mon.a 2023-07-20T02:52:05.996 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2023-07-20T02:52:05.997 DEBUG:teuthology.orchestra.run.smithi158:> sudo systemctl stop ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mon.b 2023-07-20T02:52:06.391 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:06 smithi158 systemd[1]: Stopping Ceph mon.b for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:52:06.758 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:06 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mon-b[128901]: 2023-07-20T02:52:06.488+0000 7f9ab2fb6700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:52:06.758 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:06 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mon-b[128901]: 2023-07-20T02:52:06.488+0000 7f9ab2fb6700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2023-07-20T02:52:07.141 INFO:journalctl@ceph.mon.b.smithi158.stdout:Jul 20 02:52:06 smithi158 bash[155402]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mon-b 2023-07-20T02:52:07.158 DEBUG:teuthology.orchestra.run.smithi158:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mon.b.service' 2023-07-20T02:52:07.210 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:52:07.211 INFO:tasks.cephadm.mon.c:Stopped mon.b 2023-07-20T02:52:07.211 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2023-07-20T02:52:07.212 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mon.c 2023-07-20T02:52:07.592 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:07 smithi188 systemd[1]: Stopping Ceph mon.c for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:52:07.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:07 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mon-c[129529]: 2023-07-20T02:52:07.576+0000 7fb254b21700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:52:07.964 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:07 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mon-c[129529]: 2023-07-20T02:52:07.576+0000 7fb254b21700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2023-07-20T02:52:08.309 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 20 02:52:08 smithi188 bash[157291]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mon-c 2023-07-20T02:52:08.316 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mon.c.service' 2023-07-20T02:52:08.365 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:52:08.365 INFO:tasks.cephadm.mon.c:Stopped mon.c 2023-07-20T02:52:08.366 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2023-07-20T02:52:08.366 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mgr.x 2023-07-20T02:52:08.699 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:52:08 smithi007 systemd[1]: Stopping Ceph mgr.x for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:52:09.181 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:52:08 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:52:08.888+0000 7f61496e7700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:52:09.181 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:52:08 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x[126599]: 2023-07-20T02:52:08.888+0000 7f61496e7700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-07-20T02:52:09.449 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Jul 20 02:52:09 smithi007 bash[165774]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-x 2023-07-20T02:52:09.713 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mgr.x.service' 2023-07-20T02:52:09.806 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:52:09.807 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2023-07-20T02:52:09.807 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2023-07-20T02:52:09.808 DEBUG:teuthology.orchestra.run.smithi158:> sudo systemctl stop ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mgr.y 2023-07-20T02:52:10.141 INFO:journalctl@ceph.mgr.y.smithi158.stdout:Jul 20 02:52:09 smithi158 systemd[1]: Stopping Ceph mgr.y for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:52:10.518 INFO:journalctl@ceph.mgr.y.smithi158.stdout:Jul 20 02:52:10 smithi158 bash[155700]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-y 2023-07-20T02:52:10.518 INFO:journalctl@ceph.mgr.y.smithi158.stdout:Jul 20 02:52:10 smithi158 systemd[1]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mgr.y.service: Main process exited, code=exited, status=143/n/a 2023-07-20T02:52:10.703 DEBUG:teuthology.orchestra.run.smithi158:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mgr.y.service' 2023-07-20T02:52:10.741 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:52:10.742 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2023-07-20T02:52:10.742 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2023-07-20T02:52:10.743 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mgr.z 2023-07-20T02:52:11.172 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 20 02:52:10 smithi188 systemd[1]: Stopping Ceph mgr.z for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:52:11.453 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 20 02:52:11 smithi188 bash[157459]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mgr-z 2023-07-20T02:52:11.633 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 20 02:52:11 smithi188 systemd[1]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mgr.z.service: Main process exited, code=exited, status=143/n/a 2023-07-20T02:52:11.636 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mgr.z.service' 2023-07-20T02:52:11.674 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:52:11.675 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2023-07-20T02:52:11.676 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2023-07-20T02:52:11.677 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.0 2023-07-20T02:52:12.199 INFO:journalctl@ceph.osd.0.smithi007.stdout:Jul 20 02:52:11 smithi007 systemd[1]: Stopping Ceph osd.0 for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:52:12.699 INFO:journalctl@ceph.osd.0.smithi007.stdout:Jul 20 02:52:12 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-0[139430]: 2023-07-20T02:52:12.209+0000 7fa576344700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:52:12.699 INFO:journalctl@ceph.osd.0.smithi007.stdout:Jul 20 02:52:12 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-0[139430]: 2023-07-20T02:52:12.209+0000 7fa576344700 -1 osd.0 89 *** Got signal Terminated *** 2023-07-20T02:52:12.699 INFO:journalctl@ceph.osd.0.smithi007.stdout:Jul 20 02:52:12 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-0[139430]: 2023-07-20T02:52:12.209+0000 7fa576344700 -1 osd.0 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T02:52:18.261 INFO:journalctl@ceph.osd.0.smithi007.stdout:Jul 20 02:52:18 smithi007 bash[165923]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-0 2023-07-20T02:52:20.010 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.0.service' 2023-07-20T02:52:20.103 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:52:20.103 INFO:tasks.cephadm.osd.0:Stopped osd.0 2023-07-20T02:52:20.104 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2023-07-20T02:52:20.105 DEBUG:teuthology.orchestra.run.smithi158:> sudo systemctl stop ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.1 2023-07-20T02:52:20.391 INFO:journalctl@ceph.osd.1.smithi158.stdout:Jul 20 02:52:20 smithi158 systemd[1]: Stopping Ceph osd.1 for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:52:20.891 INFO:journalctl@ceph.osd.1.smithi158.stdout:Jul 20 02:52:20 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-1[134008]: 2023-07-20T02:52:20.610+0000 7fbbd948f700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:52:20.891 INFO:journalctl@ceph.osd.1.smithi158.stdout:Jul 20 02:52:20 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-1[134008]: 2023-07-20T02:52:20.610+0000 7fbbd948f700 -1 osd.1 89 *** Got signal Terminated *** 2023-07-20T02:52:20.891 INFO:journalctl@ceph.osd.1.smithi158.stdout:Jul 20 02:52:20 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-1[134008]: 2023-07-20T02:52:20.610+0000 7fbbd948f700 -1 osd.1 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T02:52:26.642 INFO:journalctl@ceph.osd.1.smithi158.stdout:Jul 20 02:52:26 smithi158 bash[155855]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-1 2023-07-20T02:52:27.728 DEBUG:teuthology.orchestra.run.smithi158:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.1.service' 2023-07-20T02:52:27.767 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:52:27.768 INFO:tasks.cephadm.osd.1:Stopped osd.1 2023-07-20T02:52:27.768 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2023-07-20T02:52:27.769 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.2 2023-07-20T02:52:28.170 INFO:journalctl@ceph.osd.2.smithi188.stdout:Jul 20 02:52:27 smithi188 systemd[1]: Stopping Ceph osd.2 for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:52:28.464 INFO:journalctl@ceph.osd.2.smithi188.stdout:Jul 20 02:52:28 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-2[135329]: 2023-07-20T02:52:28.156+0000 7fc133140700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:52:28.464 INFO:journalctl@ceph.osd.2.smithi188.stdout:Jul 20 02:52:28 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-2[135329]: 2023-07-20T02:52:28.156+0000 7fc133140700 -1 osd.2 89 *** Got signal Terminated *** 2023-07-20T02:52:28.464 INFO:journalctl@ceph.osd.2.smithi188.stdout:Jul 20 02:52:28 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-2[135329]: 2023-07-20T02:52:28.156+0000 7fc133140700 -1 osd.2 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T02:52:33.910 INFO:journalctl@ceph.osd.2.smithi188.stdout:Jul 20 02:52:33 smithi188 bash[157604]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-2 2023-07-20T02:52:35.275 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.2.service' 2023-07-20T02:52:35.316 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:52:35.354 INFO:tasks.cephadm.osd.2:Stopped osd.2 2023-07-20T02:52:35.355 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2023-07-20T02:52:35.356 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.3 2023-07-20T02:52:35.699 INFO:journalctl@ceph.osd.3.smithi007.stdout:Jul 20 02:52:35 smithi007 systemd[1]: Stopping Ceph osd.3 for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:52:36.199 INFO:journalctl@ceph.osd.3.smithi007.stdout:Jul 20 02:52:35 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-3[145003]: 2023-07-20T02:52:35.864+0000 7f639eaa5700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:52:36.199 INFO:journalctl@ceph.osd.3.smithi007.stdout:Jul 20 02:52:35 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-3[145003]: 2023-07-20T02:52:35.864+0000 7f639eaa5700 -1 osd.3 89 *** Got signal Terminated *** 2023-07-20T02:52:36.199 INFO:journalctl@ceph.osd.3.smithi007.stdout:Jul 20 02:52:35 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-3[145003]: 2023-07-20T02:52:35.864+0000 7f639eaa5700 -1 osd.3 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T02:52:38.641 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:38 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:38.170+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:52:38.714 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:52:38 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5[140890]: 2023-07-20T02:52:38.296+0000 7fb7564cb700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.891315+0000 front 2023-07-20T02:52:16.891334+0000 (oldest deadline 2023-07-20T02:52:37.991402+0000) 2023-07-20T02:52:39.449 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:38 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:38.962+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.878406+0000 front 2023-07-20T02:52:16.878329+0000 (oldest deadline 2023-07-20T02:52:37.978433+0000) 2023-07-20T02:52:39.464 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:52:39 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5[140890]: 2023-07-20T02:52:39.326+0000 7fb7564cb700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.891315+0000 front 2023-07-20T02:52:16.891334+0000 (oldest deadline 2023-07-20T02:52:37.991402+0000) 2023-07-20T02:52:39.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:39 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:39.135+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:52:39.641 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:39 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:39.208+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:52:40.109 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:39 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:39.810+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:52:40.199 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:39 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:39.986+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.878406+0000 front 2023-07-20T02:52:16.878329+0000 (oldest deadline 2023-07-20T02:52:37.978433+0000) 2023-07-20T02:52:40.199 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:39 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:39.783+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:52:40.464 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:52:40 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5[140890]: 2023-07-20T02:52:40.282+0000 7fb7564cb700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.891315+0000 front 2023-07-20T02:52:16.891334+0000 (oldest deadline 2023-07-20T02:52:37.991402+0000) 2023-07-20T02:52:40.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:40 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:40.096+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:52:40.641 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:40 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:40.192+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:52:41.075 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:40 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:40.799+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:52:41.199 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:41 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:40.998+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.878406+0000 front 2023-07-20T02:52:16.878329+0000 (oldest deadline 2023-07-20T02:52:37.978433+0000) 2023-07-20T02:52:41.200 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:40 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:40.795+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:52:41.464 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:52:41 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5[140890]: 2023-07-20T02:52:41.265+0000 7fb7564cb700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.891315+0000 front 2023-07-20T02:52:16.891334+0000 (oldest deadline 2023-07-20T02:52:37.991402+0000) 2023-07-20T02:52:41.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:41 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:41.062+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:52:41.641 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:41 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:41.158+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:52:41.950 INFO:journalctl@ceph.osd.3.smithi007.stdout:Jul 20 02:52:41 smithi007 bash[166200]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-3 2023-07-20T02:52:41.951 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:41 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:41.759+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:52:42.103 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:41 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:41.836+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:52:42.454 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:42 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:42.045+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.878406+0000 front 2023-07-20T02:52:16.878329+0000 (oldest deadline 2023-07-20T02:52:37.978433+0000) 2023-07-20T02:52:42.464 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:52:42 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5[140890]: 2023-07-20T02:52:42.233+0000 7fb7564cb700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.891315+0000 front 2023-07-20T02:52:16.891334+0000 (oldest deadline 2023-07-20T02:52:37.991402+0000) 2023-07-20T02:52:42.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:42 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:42.090+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:52:42.641 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:42 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:42.134+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:52:42.949 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:42 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:42.719+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:52:43.118 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:42 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:42.801+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:52:43.449 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:43 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:43.047+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.878406+0000 front 2023-07-20T02:52:16.878329+0000 (oldest deadline 2023-07-20T02:52:37.978433+0000) 2023-07-20T02:52:43.464 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:52:43 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5[140890]: 2023-07-20T02:52:43.256+0000 7fb7564cb700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.891315+0000 front 2023-07-20T02:52:16.891334+0000 (oldest deadline 2023-07-20T02:52:37.991402+0000) 2023-07-20T02:52:43.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:43 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:43.104+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:52:43.584 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.3.service' 2023-07-20T02:52:43.641 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:43 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:43.132+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:52:43.676 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:52:43.676 INFO:tasks.cephadm.osd.3:Stopped osd.3 2023-07-20T02:52:43.677 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2023-07-20T02:52:43.677 DEBUG:teuthology.orchestra.run.smithi158:> sudo systemctl stop ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.4 2023-07-20T02:52:43.949 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:43 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:43.746+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:52:44.036 INFO:journalctl@ceph.osd.4.smithi158.stdout:Jul 20 02:52:43 smithi158 systemd[1]: Stopping Ceph osd.4 for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:52:44.214 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:44 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:44.060+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:52:44.215 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:43 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:43.839+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:52:44.391 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:44 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:44.095+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:52:44.392 INFO:journalctl@ceph.osd.4.smithi158.stdout:Jul 20 02:52:44 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-4[139474]: 2023-07-20T02:52:44.022+0000 7f2e4eb9c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:52:44.392 INFO:journalctl@ceph.osd.4.smithi158.stdout:Jul 20 02:52:44 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-4[139474]: 2023-07-20T02:52:44.022+0000 7f2e4eb9c700 -1 osd.4 89 *** Got signal Terminated *** 2023-07-20T02:52:44.392 INFO:journalctl@ceph.osd.4.smithi158.stdout:Jul 20 02:52:44 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-4[139474]: 2023-07-20T02:52:44.022+0000 7f2e4eb9c700 -1 osd.4 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T02:52:44.449 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:44 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:44.089+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.878406+0000 front 2023-07-20T02:52:16.878329+0000 (oldest deadline 2023-07-20T02:52:37.978433+0000) 2023-07-20T02:52:44.714 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:52:44 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5[140890]: 2023-07-20T02:52:44.222+0000 7fb7564cb700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.891315+0000 front 2023-07-20T02:52:16.891334+0000 (oldest deadline 2023-07-20T02:52:37.991402+0000) 2023-07-20T02:52:45.095 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:44 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:44.731+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:52:45.114 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:44 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:44.813+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:52:45.391 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:45 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:45.046+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:52:45.449 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:45 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:45.079+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.878406+0000 front 2023-07-20T02:52:16.878329+0000 (oldest deadline 2023-07-20T02:52:37.978433+0000) 2023-07-20T02:52:45.464 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:52:45 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5[140890]: 2023-07-20T02:52:45.257+0000 7fb7564cb700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.891315+0000 front 2023-07-20T02:52:16.891334+0000 (oldest deadline 2023-07-20T02:52:37.991402+0000) 2023-07-20T02:52:45.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:45 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:45.101+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:52:46.090 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:45 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:45.772+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:52:46.133 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:45 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:45.842+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:52:46.391 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:46 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:46.079+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:52:46.449 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:46 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:46.076+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.878406+0000 front 2023-07-20T02:52:16.878329+0000 (oldest deadline 2023-07-20T02:52:37.978433+0000) 2023-07-20T02:52:46.464 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:52:46 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5[140890]: 2023-07-20T02:52:46.291+0000 7fb7564cb700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.891315+0000 front 2023-07-20T02:52:16.891334+0000 (oldest deadline 2023-07-20T02:52:37.991402+0000) 2023-07-20T02:52:46.464 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:46 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:46.119+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:52:47.074 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:46 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:46.791+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:52:47.074 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:46 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:46.791+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:52:47.131 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:46 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:46.861+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:52:47.391 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:47 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:47.094+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:52:47.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:47 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:47.094+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:52:47.449 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:47 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:47.059+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.878406+0000 front 2023-07-20T02:52:16.878329+0000 (oldest deadline 2023-07-20T02:52:37.978433+0000) 2023-07-20T02:52:47.464 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:52:47 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5[140890]: 2023-07-20T02:52:47.320+0000 7fb7564cb700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.891315+0000 front 2023-07-20T02:52:16.891334+0000 (oldest deadline 2023-07-20T02:52:37.991402+0000) 2023-07-20T02:52:47.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:47 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:47.118+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:52:48.107 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:52:47 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:52:47.732+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.563179+0000 front 2023-07-20T02:52:21.563231+0000 (oldest deadline 2023-07-20T02:52:47.463400+0000) 2023-07-20T02:52:48.163 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:47 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:47.858+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:52:48.199 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:48 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:48.071+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.878406+0000 front 2023-07-20T02:52:16.878329+0000 (oldest deadline 2023-07-20T02:52:37.978433+0000) 2023-07-20T02:52:48.199 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:48 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:48.071+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:24.280746+0000 front 2023-07-20T02:52:24.280785+0000 (oldest deadline 2023-07-20T02:52:47.780843+0000) 2023-07-20T02:52:48.200 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:47 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:47.823+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:52:48.200 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:47 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:47.823+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:52:48.391 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:48 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:48.094+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:52:48.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:48 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:48.094+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:52:48.464 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:52:48 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5[140890]: 2023-07-20T02:52:48.290+0000 7fb7564cb700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.891315+0000 front 2023-07-20T02:52:16.891334+0000 (oldest deadline 2023-07-20T02:52:37.991402+0000) 2023-07-20T02:52:48.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:48 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:48.150+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:52:48.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:48 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:48.150+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:52:49.078 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:52:48 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:52:48.746+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.563179+0000 front 2023-07-20T02:52:21.563231+0000 (oldest deadline 2023-07-20T02:52:47.463400+0000) 2023-07-20T02:52:49.177 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:48 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:48.809+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:52:49.199 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:49 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:49.049+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.878406+0000 front 2023-07-20T02:52:16.878329+0000 (oldest deadline 2023-07-20T02:52:37.978433+0000) 2023-07-20T02:52:49.199 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:49 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:49.049+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:24.280746+0000 front 2023-07-20T02:52:24.280785+0000 (oldest deadline 2023-07-20T02:52:47.780843+0000) 2023-07-20T02:52:49.200 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:48 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:48.812+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:52:49.200 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:48 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:48.812+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:52:49.391 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:49 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:49.064+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:52:49.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:49 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:49.064+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:52:49.464 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:52:49 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5[140890]: 2023-07-20T02:52:49.336+0000 7fb7564cb700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.891315+0000 front 2023-07-20T02:52:16.891334+0000 (oldest deadline 2023-07-20T02:52:37.991402+0000) 2023-07-20T02:52:49.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:49 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:49.162+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:52:49.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:49 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:49.162+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:52:50.091 INFO:journalctl@ceph.osd.4.smithi158.stdout:Jul 20 02:52:49 smithi158 bash[156128]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-4 2023-07-20T02:52:50.091 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:52:49 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:52:49.761+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.563179+0000 front 2023-07-20T02:52:21.563231+0000 (oldest deadline 2023-07-20T02:52:47.463400+0000) 2023-07-20T02:52:50.199 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:50 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:50.033+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.878406+0000 front 2023-07-20T02:52:16.878329+0000 (oldest deadline 2023-07-20T02:52:37.978433+0000) 2023-07-20T02:52:50.199 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:50 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:50.033+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:24.280746+0000 front 2023-07-20T02:52:24.280785+0000 (oldest deadline 2023-07-20T02:52:47.780843+0000) 2023-07-20T02:52:50.200 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:49 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:49.820+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:52:50.200 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:49 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:49.820+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:52:50.214 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:49 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:49.798+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:52:50.370 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:50 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:50.076+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:52:50.371 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:50 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:50.076+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:52:50.714 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:52:50 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5[140890]: 2023-07-20T02:52:50.363+0000 7fb7564cb700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.891315+0000 front 2023-07-20T02:52:16.891334+0000 (oldest deadline 2023-07-20T02:52:37.991402+0000) 2023-07-20T02:52:50.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:50 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:50.203+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:52:50.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:50 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:50.203+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:52:50.972 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:52:50 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:52:50.726+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.563179+0000 front 2023-07-20T02:52:21.563231+0000 (oldest deadline 2023-07-20T02:52:47.463400+0000) 2023-07-20T02:52:50.978 DEBUG:teuthology.orchestra.run.smithi158:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.4.service' 2023-07-20T02:52:51.020 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:52:51.021 INFO:tasks.cephadm.osd.4:Stopped osd.4 2023-07-20T02:52:51.021 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2023-07-20T02:52:51.022 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.5 2023-07-20T02:52:51.184 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:52:51 smithi188 systemd[1]: Stopping Ceph osd.5 for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:52:51.185 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:50 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:50.804+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:52:51.185 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:50 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:50.804+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:23.180901+0000 front 2023-07-20T02:52:23.180880+0000 (oldest deadline 2023-07-20T02:52:50.781771+0000) 2023-07-20T02:52:51.199 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:50 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:50.796+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:52:51.200 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:50 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:50.796+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:52:51.200 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:51 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:51.041+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.878406+0000 front 2023-07-20T02:52:16.878329+0000 (oldest deadline 2023-07-20T02:52:37.978433+0000) 2023-07-20T02:52:51.200 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:51 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:51.041+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:24.280746+0000 front 2023-07-20T02:52:24.280785+0000 (oldest deadline 2023-07-20T02:52:47.780843+0000) 2023-07-20T02:52:51.391 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:51 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:51.032+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:52:51.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:51 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:51.032+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:52:51.464 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:51 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:51.169+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:52:51.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:51 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:51.169+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:52:51.465 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:52:51 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5[140890]: 2023-07-20T02:52:51.374+0000 7fb75bed8700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:52:51.465 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:52:51 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5[140890]: 2023-07-20T02:52:51.374+0000 7fb75bed8700 -1 osd.5 89 *** Got signal Terminated *** 2023-07-20T02:52:51.465 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:52:51 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5[140890]: 2023-07-20T02:52:51.374+0000 7fb75bed8700 -1 osd.5 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T02:52:51.465 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:52:51 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5[140890]: 2023-07-20T02:52:51.403+0000 7fb7564cb700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.891315+0000 front 2023-07-20T02:52:16.891334+0000 (oldest deadline 2023-07-20T02:52:37.991402+0000) 2023-07-20T02:52:51.466 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:52:51 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5[140890]: 2023-07-20T02:52:51.403+0000 7fb7564cb700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.493534+0000 front 2023-07-20T02:52:25.493756+0000 (oldest deadline 2023-07-20T02:52:51.393674+0000) 2023-07-20T02:52:52.049 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:51 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:51.765+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:52:52.050 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:51 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:51.765+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:52:52.053 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:52:51 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:52:51.725+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.563179+0000 front 2023-07-20T02:52:21.563231+0000 (oldest deadline 2023-07-20T02:52:47.463400+0000) 2023-07-20T02:52:52.207 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:51 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:51.834+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:52:52.208 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:51 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:51.834+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:23.180901+0000 front 2023-07-20T02:52:23.180880+0000 (oldest deadline 2023-07-20T02:52:50.781771+0000) 2023-07-20T02:52:52.391 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:52 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:52.039+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:52:52.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:52 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:52.039+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:52:52.449 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:52 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:52.035+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.878406+0000 front 2023-07-20T02:52:16.878329+0000 (oldest deadline 2023-07-20T02:52:37.978433+0000) 2023-07-20T02:52:52.450 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:52 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:52.035+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:24.280746+0000 front 2023-07-20T02:52:24.280785+0000 (oldest deadline 2023-07-20T02:52:47.780843+0000) 2023-07-20T02:52:52.464 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:52:52 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5[140890]: 2023-07-20T02:52:52.374+0000 7fb7564cb700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.891315+0000 front 2023-07-20T02:52:16.891334+0000 (oldest deadline 2023-07-20T02:52:37.991402+0000) 2023-07-20T02:52:52.465 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:52:52 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5[140890]: 2023-07-20T02:52:52.374+0000 7fb7564cb700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.493534+0000 front 2023-07-20T02:52:25.493756+0000 (oldest deadline 2023-07-20T02:52:51.393674+0000) 2023-07-20T02:52:52.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:52 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:52.193+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:52:52.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:52 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:52.193+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:52:53.039 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:52:52 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:52:52.752+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.563179+0000 front 2023-07-20T02:52:21.563231+0000 (oldest deadline 2023-07-20T02:52:47.463400+0000) 2023-07-20T02:52:53.049 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:52 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:52.749+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:52:53.050 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:52 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:52.749+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:52:53.158 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:52 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:52.800+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:52:53.158 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:52 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:52.800+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:23.180901+0000 front 2023-07-20T02:52:23.180880+0000 (oldest deadline 2023-07-20T02:52:50.781771+0000) 2023-07-20T02:52:53.391 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:53 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:53.026+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:52:53.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:53 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:53.026+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:52:53.449 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:53 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:53.035+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.878406+0000 front 2023-07-20T02:52:16.878329+0000 (oldest deadline 2023-07-20T02:52:37.978433+0000) 2023-07-20T02:52:53.449 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:53 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:53.035+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:24.280746+0000 front 2023-07-20T02:52:24.280785+0000 (oldest deadline 2023-07-20T02:52:47.780843+0000) 2023-07-20T02:52:53.464 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:53 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:53.143+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:52:53.464 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:53 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:53.143+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:52:53.465 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:52:53 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5[140890]: 2023-07-20T02:52:53.382+0000 7fb7564cb700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.891315+0000 front 2023-07-20T02:52:16.891334+0000 (oldest deadline 2023-07-20T02:52:37.991402+0000) 2023-07-20T02:52:53.465 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:52:53 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5[140890]: 2023-07-20T02:52:53.382+0000 7fb7564cb700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.493534+0000 front 2023-07-20T02:52:25.493756+0000 (oldest deadline 2023-07-20T02:52:51.393674+0000) 2023-07-20T02:52:54.006 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:53 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:53.732+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:52:54.006 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:53 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:53.732+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:52:54.052 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:52:53 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:52:53.728+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.563179+0000 front 2023-07-20T02:52:21.563231+0000 (oldest deadline 2023-07-20T02:52:47.463400+0000) 2023-07-20T02:52:54.205 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:53 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:53.838+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:52:54.205 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:53 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:53.838+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:23.180901+0000 front 2023-07-20T02:52:23.180880+0000 (oldest deadline 2023-07-20T02:52:50.781771+0000) 2023-07-20T02:52:54.391 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:54 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:54.038+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:52:54.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:54 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:54.038+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:52:54.449 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:53 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:53.991+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.878406+0000 front 2023-07-20T02:52:16.878329+0000 (oldest deadline 2023-07-20T02:52:37.978433+0000) 2023-07-20T02:52:54.450 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:53 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:53.991+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:24.280746+0000 front 2023-07-20T02:52:24.280785+0000 (oldest deadline 2023-07-20T02:52:47.780843+0000) 2023-07-20T02:52:54.464 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:52:54 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5[140890]: 2023-07-20T02:52:54.357+0000 7fb7564cb700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.891315+0000 front 2023-07-20T02:52:16.891334+0000 (oldest deadline 2023-07-20T02:52:37.991402+0000) 2023-07-20T02:52:54.465 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:52:54 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5[140890]: 2023-07-20T02:52:54.357+0000 7fb7564cb700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.493534+0000 front 2023-07-20T02:52:25.493756+0000 (oldest deadline 2023-07-20T02:52:51.393674+0000) 2023-07-20T02:52:54.465 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:52:54 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5[140890]: 2023-07-20T02:52:54.357+0000 7fb7564cb700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:31.394021+0000 front 2023-07-20T02:52:31.394046+0000 (oldest deadline 2023-07-20T02:52:53.694315+0000) 2023-07-20T02:52:54.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:54 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:54.190+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:52:54.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:54 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:54.190+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:52:54.992 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:54 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:54.708+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:52:54.992 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:54 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:54.708+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:52:55.073 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:52:54 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:52:54.748+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.563179+0000 front 2023-07-20T02:52:21.563231+0000 (oldest deadline 2023-07-20T02:52:47.463400+0000) 2023-07-20T02:52:55.073 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:52:54 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:52:54.748+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:32.164343+0000 front 2023-07-20T02:52:32.164405+0000 (oldest deadline 2023-07-20T02:52:54.464409+0000) 2023-07-20T02:52:55.194 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:54 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:54.864+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:52:55.195 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:54 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:54.864+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:23.180901+0000 front 2023-07-20T02:52:23.180880+0000 (oldest deadline 2023-07-20T02:52:50.781771+0000) 2023-07-20T02:52:55.195 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:54 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:54.864+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.782398+0000 front 2023-07-20T02:52:30.782308+0000 (oldest deadline 2023-07-20T02:52:54.282509+0000) 2023-07-20T02:52:55.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:55 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:55.058+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:52:55.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:55 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:55.058+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:52:55.449 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:54 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:54.977+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.878406+0000 front 2023-07-20T02:52:16.878329+0000 (oldest deadline 2023-07-20T02:52:37.978433+0000) 2023-07-20T02:52:55.450 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:54 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:54.977+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:24.280746+0000 front 2023-07-20T02:52:24.280785+0000 (oldest deadline 2023-07-20T02:52:47.780843+0000) 2023-07-20T02:52:55.464 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:52:55 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5[140890]: 2023-07-20T02:52:55.358+0000 7fb7564cb700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.891315+0000 front 2023-07-20T02:52:16.891334+0000 (oldest deadline 2023-07-20T02:52:37.991402+0000) 2023-07-20T02:52:55.465 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:52:55 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5[140890]: 2023-07-20T02:52:55.358+0000 7fb7564cb700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.493534+0000 front 2023-07-20T02:52:25.493756+0000 (oldest deadline 2023-07-20T02:52:51.393674+0000) 2023-07-20T02:52:55.465 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:52:55 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5[140890]: 2023-07-20T02:52:55.358+0000 7fb7564cb700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:31.394021+0000 front 2023-07-20T02:52:31.394046+0000 (oldest deadline 2023-07-20T02:52:53.694315+0000) 2023-07-20T02:52:55.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:55 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:55.179+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:52:55.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:55 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:55.179+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:52:55.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:55 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:55.179+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:52:56.119 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:52:55 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:52:55.705+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.563179+0000 front 2023-07-20T02:52:21.563231+0000 (oldest deadline 2023-07-20T02:52:47.463400+0000) 2023-07-20T02:52:56.120 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:52:55 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:52:55.705+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:32.164343+0000 front 2023-07-20T02:52:32.164405+0000 (oldest deadline 2023-07-20T02:52:54.464409+0000) 2023-07-20T02:52:56.145 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:55 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:55.861+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:52:56.145 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:55 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:55.861+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:23.180901+0000 front 2023-07-20T02:52:23.180880+0000 (oldest deadline 2023-07-20T02:52:50.781771+0000) 2023-07-20T02:52:56.145 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:55 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:55.861+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.782398+0000 front 2023-07-20T02:52:30.782308+0000 (oldest deadline 2023-07-20T02:52:54.282509+0000) 2023-07-20T02:52:56.199 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:55 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:55.945+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.878406+0000 front 2023-07-20T02:52:16.878329+0000 (oldest deadline 2023-07-20T02:52:37.978433+0000) 2023-07-20T02:52:56.199 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:55 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:55.945+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:24.280746+0000 front 2023-07-20T02:52:24.280785+0000 (oldest deadline 2023-07-20T02:52:47.780843+0000) 2023-07-20T02:52:56.200 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:55 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:55.757+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:52:56.200 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:55 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:55.757+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:52:56.200 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:55 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:55.757+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:29.391927+0000 front 2023-07-20T02:52:29.391909+0000 (oldest deadline 2023-07-20T02:52:55.292047+0000) 2023-07-20T02:52:56.391 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:56 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:56.104+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:52:56.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:56 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:56.104+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:52:56.464 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:52:56 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5[140890]: 2023-07-20T02:52:56.336+0000 7fb7564cb700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.891315+0000 front 2023-07-20T02:52:16.891334+0000 (oldest deadline 2023-07-20T02:52:37.991402+0000) 2023-07-20T02:52:56.465 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:52:56 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5[140890]: 2023-07-20T02:52:56.336+0000 7fb7564cb700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.493534+0000 front 2023-07-20T02:52:25.493756+0000 (oldest deadline 2023-07-20T02:52:51.393674+0000) 2023-07-20T02:52:56.465 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:52:56 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5[140890]: 2023-07-20T02:52:56.336+0000 7fb7564cb700 -1 osd.5 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:31.394021+0000 front 2023-07-20T02:52:31.394046+0000 (oldest deadline 2023-07-20T02:52:53.694315+0000) 2023-07-20T02:52:56.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:56 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:56.130+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:52:56.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:56 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:56.130+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:52:56.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:56 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:56.130+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:52:57.062 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 20 02:52:56 smithi188 bash[157876]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-5 2023-07-20T02:52:57.063 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:56 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:56.819+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:52:57.063 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:56 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:56.819+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:23.180901+0000 front 2023-07-20T02:52:23.180880+0000 (oldest deadline 2023-07-20T02:52:50.781771+0000) 2023-07-20T02:52:57.063 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:56 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:56.819+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.782398+0000 front 2023-07-20T02:52:30.782308+0000 (oldest deadline 2023-07-20T02:52:54.282509+0000) 2023-07-20T02:52:57.142 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:52:56 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:52:56.693+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.563179+0000 front 2023-07-20T02:52:21.563231+0000 (oldest deadline 2023-07-20T02:52:47.463400+0000) 2023-07-20T02:52:57.142 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:52:56 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:52:56.693+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:32.164343+0000 front 2023-07-20T02:52:32.164405+0000 (oldest deadline 2023-07-20T02:52:54.464409+0000) 2023-07-20T02:52:57.199 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:56 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:56.912+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.878406+0000 front 2023-07-20T02:52:16.878329+0000 (oldest deadline 2023-07-20T02:52:37.978433+0000) 2023-07-20T02:52:57.200 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:56 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:56.912+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:24.280746+0000 front 2023-07-20T02:52:24.280785+0000 (oldest deadline 2023-07-20T02:52:47.780843+0000) 2023-07-20T02:52:57.200 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:56 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:56.912+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:32.482438+0000 front 2023-07-20T02:52:32.482413+0000 (oldest deadline 2023-07-20T02:52:56.582446+0000) 2023-07-20T02:52:57.200 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:56 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:56.787+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:52:57.200 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:56 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:56.787+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:52:57.200 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:56 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:56.787+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:29.391927+0000 front 2023-07-20T02:52:29.391909+0000 (oldest deadline 2023-07-20T02:52:55.292047+0000) 2023-07-20T02:52:57.464 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:57 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:57.132+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:52:57.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:57 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:57.132+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:52:57.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:57 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:57.132+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:52:57.641 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:57 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:57.135+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:52:57.642 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:57 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:57.135+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:52:57.964 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:57 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:57.803+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:52:57.965 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:57 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:57.803+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:23.180901+0000 front 2023-07-20T02:52:23.180880+0000 (oldest deadline 2023-07-20T02:52:50.781771+0000) 2023-07-20T02:52:57.965 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:57 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:57.803+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.782398+0000 front 2023-07-20T02:52:30.782308+0000 (oldest deadline 2023-07-20T02:52:54.282509+0000) 2023-07-20T02:52:58.141 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:52:57 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:52:57.725+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.563179+0000 front 2023-07-20T02:52:21.563231+0000 (oldest deadline 2023-07-20T02:52:47.463400+0000) 2023-07-20T02:52:58.142 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:52:57 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:52:57.725+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:32.164343+0000 front 2023-07-20T02:52:32.164405+0000 (oldest deadline 2023-07-20T02:52:54.464409+0000) 2023-07-20T02:52:58.199 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:57 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:57.924+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.878406+0000 front 2023-07-20T02:52:16.878329+0000 (oldest deadline 2023-07-20T02:52:37.978433+0000) 2023-07-20T02:52:58.200 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:57 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:57.924+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:24.280746+0000 front 2023-07-20T02:52:24.280785+0000 (oldest deadline 2023-07-20T02:52:47.780843+0000) 2023-07-20T02:52:58.200 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:57 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:57.924+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:32.482438+0000 front 2023-07-20T02:52:32.482413+0000 (oldest deadline 2023-07-20T02:52:56.582446+0000) 2023-07-20T02:52:58.200 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:57 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:57.791+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:52:58.200 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:57 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:57.791+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:52:58.200 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:57 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:57.791+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:29.391927+0000 front 2023-07-20T02:52:29.391909+0000 (oldest deadline 2023-07-20T02:52:55.292047+0000) 2023-07-20T02:52:58.420 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:58 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:58.112+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:52:58.421 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:58 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:58.112+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:52:58.421 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:58 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:58.112+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:52:58.429 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.5.service' 2023-07-20T02:52:58.480 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:52:58.481 INFO:tasks.cephadm.osd.5:Stopped osd.5 2023-07-20T02:52:58.482 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2023-07-20T02:52:58.482 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.6 2023-07-20T02:52:58.642 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:58 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:58.142+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:52:58.642 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:58 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:58.142+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:52:58.851 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:58 smithi007 systemd[1]: Stopping Ceph osd.6 for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:52:59.124 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:58 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:58.930+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.878406+0000 front 2023-07-20T02:52:16.878329+0000 (oldest deadline 2023-07-20T02:52:37.978433+0000) 2023-07-20T02:52:59.124 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:58 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:58.930+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:24.280746+0000 front 2023-07-20T02:52:24.280785+0000 (oldest deadline 2023-07-20T02:52:47.780843+0000) 2023-07-20T02:52:59.124 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:58 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:58.930+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:32.482438+0000 front 2023-07-20T02:52:32.482413+0000 (oldest deadline 2023-07-20T02:52:56.582446+0000) 2023-07-20T02:52:59.125 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:58 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:58.837+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:52:59.125 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:58 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:58.837+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:52:59.125 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:58 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:58.837+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:29.391927+0000 front 2023-07-20T02:52:29.391909+0000 (oldest deadline 2023-07-20T02:52:55.292047+0000) 2023-07-20T02:52:59.132 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:52:58 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:52:58.772+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.563179+0000 front 2023-07-20T02:52:21.563231+0000 (oldest deadline 2023-07-20T02:52:47.463400+0000) 2023-07-20T02:52:59.133 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:52:58 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:52:58.772+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:32.164343+0000 front 2023-07-20T02:52:32.164405+0000 (oldest deadline 2023-07-20T02:52:54.464409+0000) 2023-07-20T02:52:59.158 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:58 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:58.803+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:52:59.158 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:58 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:58.803+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:23.180901+0000 front 2023-07-20T02:52:23.180880+0000 (oldest deadline 2023-07-20T02:52:50.781771+0000) 2023-07-20T02:52:59.158 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:58 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:58.803+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.782398+0000 front 2023-07-20T02:52:30.782308+0000 (oldest deadline 2023-07-20T02:52:54.282509+0000) 2023-07-20T02:52:59.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:59 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:59.118+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:52:59.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:59 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:59.118+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:52:59.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:52:59 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:52:59.118+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:52:59.449 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:59 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:59.109+0000 7f7815178700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:52:59.450 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:59 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:59.109+0000 7f7815178700 -1 osd.6 89 *** Got signal Terminated *** 2023-07-20T02:52:59.450 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:59 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:59.109+0000 7f7815178700 -1 osd.6 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T02:52:59.464 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:59 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:59.143+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:52:59.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:59 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:59.143+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:52:59.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:52:59 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:52:59.143+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:00.093 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:52:59 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:52:59.799+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.563179+0000 front 2023-07-20T02:52:21.563231+0000 (oldest deadline 2023-07-20T02:52:47.463400+0000) 2023-07-20T02:53:00.093 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:52:59 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:52:59.799+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:32.164343+0000 front 2023-07-20T02:52:32.164405+0000 (oldest deadline 2023-07-20T02:52:54.464409+0000) 2023-07-20T02:53:00.145 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:59 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:59.797+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:53:00.145 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:59 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:59.797+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:23.180901+0000 front 2023-07-20T02:52:23.180880+0000 (oldest deadline 2023-07-20T02:52:50.781771+0000) 2023-07-20T02:53:00.145 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:52:59 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:52:59.797+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.782398+0000 front 2023-07-20T02:52:30.782308+0000 (oldest deadline 2023-07-20T02:52:54.282509+0000) 2023-07-20T02:53:00.199 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:59 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:59.981+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.878406+0000 front 2023-07-20T02:52:16.878329+0000 (oldest deadline 2023-07-20T02:52:37.978433+0000) 2023-07-20T02:53:00.200 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:59 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:59.981+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:24.280746+0000 front 2023-07-20T02:52:24.280785+0000 (oldest deadline 2023-07-20T02:52:47.780843+0000) 2023-07-20T02:53:00.200 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:52:59 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:52:59.981+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:32.482438+0000 front 2023-07-20T02:52:32.482413+0000 (oldest deadline 2023-07-20T02:52:56.582446+0000) 2023-07-20T02:53:00.200 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:59 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:59.885+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:53:00.200 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:59 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:59.885+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:53:00.201 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:52:59 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:52:59.885+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:29.391927+0000 front 2023-07-20T02:52:29.391909+0000 (oldest deadline 2023-07-20T02:52:55.292047+0000) 2023-07-20T02:53:00.391 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:00 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:00.078+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:00.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:00 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:00.078+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:00.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:00 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:00.078+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:00.464 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:00 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:00.131+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:00.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:00 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:00.131+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:00.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:00 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:00.131+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:01.119 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:00 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:00.767+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:53:01.119 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:00 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:00.767+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:23.180901+0000 front 2023-07-20T02:52:23.180880+0000 (oldest deadline 2023-07-20T02:52:50.781771+0000) 2023-07-20T02:53:01.119 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:00 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:00.767+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.782398+0000 front 2023-07-20T02:52:30.782308+0000 (oldest deadline 2023-07-20T02:52:54.282509+0000) 2023-07-20T02:53:01.142 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:01 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:01.057+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:01.142 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:01 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:01.057+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:01.142 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:01 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:01.057+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:01.142 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:00 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:00.818+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.563179+0000 front 2023-07-20T02:52:21.563231+0000 (oldest deadline 2023-07-20T02:52:47.463400+0000) 2023-07-20T02:53:01.143 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:00 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:00.818+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:32.164343+0000 front 2023-07-20T02:52:32.164405+0000 (oldest deadline 2023-07-20T02:52:54.464409+0000) 2023-07-20T02:53:01.199 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:53:01 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:53:01.022+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.878406+0000 front 2023-07-20T02:52:16.878329+0000 (oldest deadline 2023-07-20T02:52:37.978433+0000) 2023-07-20T02:53:01.200 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:53:01 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:53:01.022+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:24.280746+0000 front 2023-07-20T02:52:24.280785+0000 (oldest deadline 2023-07-20T02:52:47.780843+0000) 2023-07-20T02:53:01.200 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:53:01 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:53:01.022+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:32.482438+0000 front 2023-07-20T02:52:32.482413+0000 (oldest deadline 2023-07-20T02:52:56.582446+0000) 2023-07-20T02:53:01.200 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:00 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:00.901+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:53:01.200 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:00 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:00.901+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:53:01.200 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:00 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:00.901+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:29.391927+0000 front 2023-07-20T02:52:29.391909+0000 (oldest deadline 2023-07-20T02:52:55.292047+0000) 2023-07-20T02:53:01.464 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:01 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:01.105+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:01.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:01 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:01.105+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:01.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:01 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:01.105+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:02.142 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:02 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:02.011+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:02.142 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:02 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:02.011+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:02.142 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:02 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:02.011+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:02.143 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:01 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:01.860+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.563179+0000 front 2023-07-20T02:52:21.563231+0000 (oldest deadline 2023-07-20T02:52:47.463400+0000) 2023-07-20T02:53:02.143 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:01 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:01.860+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:32.164343+0000 front 2023-07-20T02:52:32.164405+0000 (oldest deadline 2023-07-20T02:52:54.464409+0000) 2023-07-20T02:53:02.143 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:01 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:01.860+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.765806+0000 front 2023-07-20T02:52:39.765630+0000 (oldest deadline 2023-07-20T02:53:01.465683+0000) 2023-07-20T02:53:02.144 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:01 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:01.759+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:53:02.144 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:01 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:01.759+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:23.180901+0000 front 2023-07-20T02:52:23.180880+0000 (oldest deadline 2023-07-20T02:52:50.781771+0000) 2023-07-20T02:53:02.144 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:01 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:01.759+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.782398+0000 front 2023-07-20T02:52:30.782308+0000 (oldest deadline 2023-07-20T02:52:54.282509+0000) 2023-07-20T02:53:02.199 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:53:02 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:53:02.039+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.878406+0000 front 2023-07-20T02:52:16.878329+0000 (oldest deadline 2023-07-20T02:52:37.978433+0000) 2023-07-20T02:53:02.200 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:53:02 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:53:02.039+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:24.280746+0000 front 2023-07-20T02:52:24.280785+0000 (oldest deadline 2023-07-20T02:52:47.780843+0000) 2023-07-20T02:53:02.200 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:53:02 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:53:02.039+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:32.482438+0000 front 2023-07-20T02:52:32.482413+0000 (oldest deadline 2023-07-20T02:52:56.582446+0000) 2023-07-20T02:53:02.200 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:01 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:01.903+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:53:02.200 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:01 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:01.903+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:53:02.200 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:01 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:01.903+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:29.391927+0000 front 2023-07-20T02:52:29.391909+0000 (oldest deadline 2023-07-20T02:52:55.292047+0000) 2023-07-20T02:53:02.464 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:02 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:02.130+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:02.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:02 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:02.130+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:02.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:02 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:02.130+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:03.142 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:02 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:02.804+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:53:03.142 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:02 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:02.804+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:23.180901+0000 front 2023-07-20T02:52:23.180880+0000 (oldest deadline 2023-07-20T02:52:50.781771+0000) 2023-07-20T02:53:03.143 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:02 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:02.804+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.782398+0000 front 2023-07-20T02:52:30.782308+0000 (oldest deadline 2023-07-20T02:52:54.282509+0000) 2023-07-20T02:53:03.143 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:02 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:02.804+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:36.583749+0000 front 2023-07-20T02:52:36.583925+0000 (oldest deadline 2023-07-20T02:53:01.883610+0000) 2023-07-20T02:53:03.144 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:03 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:03.055+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:03.144 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:03 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:03.055+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:03.144 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:03 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:03.055+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:03.144 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:02 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:02.828+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.563179+0000 front 2023-07-20T02:52:21.563231+0000 (oldest deadline 2023-07-20T02:52:47.463400+0000) 2023-07-20T02:53:03.145 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:02 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:02.828+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:32.164343+0000 front 2023-07-20T02:52:32.164405+0000 (oldest deadline 2023-07-20T02:52:54.464409+0000) 2023-07-20T02:53:03.145 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:02 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:02.828+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.765806+0000 front 2023-07-20T02:52:39.765630+0000 (oldest deadline 2023-07-20T02:53:01.465683+0000) 2023-07-20T02:53:03.449 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:53:03 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:53:03.046+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.878406+0000 front 2023-07-20T02:52:16.878329+0000 (oldest deadline 2023-07-20T02:52:37.978433+0000) 2023-07-20T02:53:03.449 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:53:03 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:53:03.046+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:24.280746+0000 front 2023-07-20T02:52:24.280785+0000 (oldest deadline 2023-07-20T02:52:47.780843+0000) 2023-07-20T02:53:03.450 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:53:03 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:53:03.046+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:32.482438+0000 front 2023-07-20T02:52:32.482413+0000 (oldest deadline 2023-07-20T02:52:56.582446+0000) 2023-07-20T02:53:03.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:02 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:02.944+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:53:03.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:02 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:02.944+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:53:03.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:02 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:02.944+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:29.391927+0000 front 2023-07-20T02:52:29.391909+0000 (oldest deadline 2023-07-20T02:52:55.292047+0000) 2023-07-20T02:53:03.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:02 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:02.944+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.393003+0000 front 2023-07-20T02:52:39.393069+0000 (oldest deadline 2023-07-20T02:53:02.193566+0000) 2023-07-20T02:53:03.464 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:03 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:03.128+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:03.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:03 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:03.128+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:03.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:03 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:03.128+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:04.089 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:03 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:03.947+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:53:04.089 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:03 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:03.947+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:53:04.089 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:03 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:03.947+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:29.391927+0000 front 2023-07-20T02:52:29.391909+0000 (oldest deadline 2023-07-20T02:52:55.292047+0000) 2023-07-20T02:53:04.090 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:03 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:03.947+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.393003+0000 front 2023-07-20T02:52:39.393069+0000 (oldest deadline 2023-07-20T02:53:02.193566+0000) 2023-07-20T02:53:04.090 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:53:04 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:53:04.068+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.878406+0000 front 2023-07-20T02:52:16.878329+0000 (oldest deadline 2023-07-20T02:52:37.978433+0000) 2023-07-20T02:53:04.091 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:53:04 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:53:04.068+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:24.280746+0000 front 2023-07-20T02:52:24.280785+0000 (oldest deadline 2023-07-20T02:52:47.780843+0000) 2023-07-20T02:53:04.091 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:53:04 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:53:04.068+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:32.482438+0000 front 2023-07-20T02:52:32.482413+0000 (oldest deadline 2023-07-20T02:52:56.582446+0000) 2023-07-20T02:53:04.142 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:04 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:04.079+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:04.142 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:04 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:04.079+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:04.142 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:04 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:04.079+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:04.142 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:04 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:04.079+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:38.634686+0000 front 2023-07-20T02:52:38.634805+0000 (oldest deadline 2023-07-20T02:53:03.934673+0000) 2023-07-20T02:53:04.143 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:03 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:03.834+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.563179+0000 front 2023-07-20T02:52:21.563231+0000 (oldest deadline 2023-07-20T02:52:47.463400+0000) 2023-07-20T02:53:04.143 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:03 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:03.834+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:32.164343+0000 front 2023-07-20T02:52:32.164405+0000 (oldest deadline 2023-07-20T02:52:54.464409+0000) 2023-07-20T02:53:04.143 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:03 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:03.834+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.765806+0000 front 2023-07-20T02:52:39.765630+0000 (oldest deadline 2023-07-20T02:53:01.465683+0000) 2023-07-20T02:53:04.215 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:04 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:04.085+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:04.215 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:04 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:04.085+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:04.215 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:04 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:04.085+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:04.215 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:03 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:03.836+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:53:04.216 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:03 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:03.836+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:23.180901+0000 front 2023-07-20T02:52:23.180880+0000 (oldest deadline 2023-07-20T02:52:50.781771+0000) 2023-07-20T02:53:04.216 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:03 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:03.836+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.782398+0000 front 2023-07-20T02:52:30.782308+0000 (oldest deadline 2023-07-20T02:52:54.282509+0000) 2023-07-20T02:53:04.216 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:03 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:03.836+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:36.583749+0000 front 2023-07-20T02:52:36.583925+0000 (oldest deadline 2023-07-20T02:53:01.883610+0000) 2023-07-20T02:53:05.102 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:04 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:04.804+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.563179+0000 front 2023-07-20T02:52:21.563231+0000 (oldest deadline 2023-07-20T02:52:47.463400+0000) 2023-07-20T02:53:05.102 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:04 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:04.804+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:32.164343+0000 front 2023-07-20T02:52:32.164405+0000 (oldest deadline 2023-07-20T02:52:54.464409+0000) 2023-07-20T02:53:05.103 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:04 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:04.804+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.765806+0000 front 2023-07-20T02:52:39.765630+0000 (oldest deadline 2023-07-20T02:53:01.465683+0000) 2023-07-20T02:53:05.145 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:04 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:04.868+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:53:05.145 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:04 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:04.868+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:23.180901+0000 front 2023-07-20T02:52:23.180880+0000 (oldest deadline 2023-07-20T02:52:50.781771+0000) 2023-07-20T02:53:05.145 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:04 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:04.868+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.782398+0000 front 2023-07-20T02:52:30.782308+0000 (oldest deadline 2023-07-20T02:52:54.282509+0000) 2023-07-20T02:53:05.145 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:04 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:04.868+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:36.583749+0000 front 2023-07-20T02:52:36.583925+0000 (oldest deadline 2023-07-20T02:53:01.883610+0000) 2023-07-20T02:53:05.199 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:53:04 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6[150453]: 2023-07-20T02:53:04.068+0000 7f781176f700 -1 osd.6 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:38.283715+0000 front 2023-07-20T02:52:38.283752+0000 (oldest deadline 2023-07-20T02:53:03.583944+0000) 2023-07-20T02:53:05.200 INFO:journalctl@ceph.osd.6.smithi007.stdout:Jul 20 02:53:04 smithi007 bash[166482]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-6 2023-07-20T02:53:05.200 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:04 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:04.941+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:53:05.200 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:04 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:04.941+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:53:05.200 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:04 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:04.941+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:29.391927+0000 front 2023-07-20T02:52:29.391909+0000 (oldest deadline 2023-07-20T02:52:55.292047+0000) 2023-07-20T02:53:05.201 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:04 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:04.941+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.393003+0000 front 2023-07-20T02:52:39.393069+0000 (oldest deadline 2023-07-20T02:53:02.193566+0000) 2023-07-20T02:53:05.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:05 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:05.088+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:05.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:05 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:05.088+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:05.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:05 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:05.088+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:05.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:05 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:05.088+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:38.634686+0000 front 2023-07-20T02:52:38.634805+0000 (oldest deadline 2023-07-20T02:53:03.934673+0000) 2023-07-20T02:53:05.464 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:05 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:05.130+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:05.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:05 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:05.130+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:05.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:05 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:05.130+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:06.142 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:06 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:06.078+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:06.142 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:06 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:06.078+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:06.142 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:06 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:06.078+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:06.142 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:06 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:06.078+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:38.634686+0000 front 2023-07-20T02:52:38.634805+0000 (oldest deadline 2023-07-20T02:53:03.934673+0000) 2023-07-20T02:53:06.142 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:05 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:05.853+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.563179+0000 front 2023-07-20T02:52:21.563231+0000 (oldest deadline 2023-07-20T02:52:47.463400+0000) 2023-07-20T02:53:06.143 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:05 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:05.853+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:32.164343+0000 front 2023-07-20T02:52:32.164405+0000 (oldest deadline 2023-07-20T02:52:54.464409+0000) 2023-07-20T02:53:06.143 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:05 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:05.853+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.765806+0000 front 2023-07-20T02:52:39.765630+0000 (oldest deadline 2023-07-20T02:53:01.465683+0000) 2023-07-20T02:53:06.193 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:05 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:05.858+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:53:06.193 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:05 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:05.858+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:23.180901+0000 front 2023-07-20T02:52:23.180880+0000 (oldest deadline 2023-07-20T02:52:50.781771+0000) 2023-07-20T02:53:06.193 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:05 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:05.858+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.782398+0000 front 2023-07-20T02:52:30.782308+0000 (oldest deadline 2023-07-20T02:52:54.282509+0000) 2023-07-20T02:53:06.193 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:05 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:05.858+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:36.583749+0000 front 2023-07-20T02:52:36.583925+0000 (oldest deadline 2023-07-20T02:53:01.883610+0000) 2023-07-20T02:53:06.199 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:05 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:05.978+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:53:06.200 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:05 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:05.978+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:53:06.200 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:05 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:05.978+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:29.391927+0000 front 2023-07-20T02:52:29.391909+0000 (oldest deadline 2023-07-20T02:52:55.292047+0000) 2023-07-20T02:53:06.200 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:05 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:05.978+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.393003+0000 front 2023-07-20T02:52:39.393069+0000 (oldest deadline 2023-07-20T02:53:02.193566+0000) 2023-07-20T02:53:06.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:06 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:06.179+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:06.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:06 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:06.179+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:06.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:06 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:06.179+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:06.702 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.6.service' 2023-07-20T02:53:06.792 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:53:06.793 INFO:tasks.cephadm.osd.6:Stopped osd.6 2023-07-20T02:53:06.794 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2023-07-20T02:53:06.795 DEBUG:teuthology.orchestra.run.smithi158:> sudo systemctl stop ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.7 2023-07-20T02:53:07.131 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:06 smithi158 systemd[1]: Stopping Ceph osd.7 for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:53:07.132 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:06 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:06.888+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.563179+0000 front 2023-07-20T02:52:21.563231+0000 (oldest deadline 2023-07-20T02:52:47.463400+0000) 2023-07-20T02:53:07.132 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:06 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:06.888+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:32.164343+0000 front 2023-07-20T02:52:32.164405+0000 (oldest deadline 2023-07-20T02:52:54.464409+0000) 2023-07-20T02:53:07.132 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:06 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:06.888+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.765806+0000 front 2023-07-20T02:52:39.765630+0000 (oldest deadline 2023-07-20T02:53:01.465683+0000) 2023-07-20T02:53:07.202 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:06 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:06.842+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:53:07.203 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:06 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:06.842+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:23.180901+0000 front 2023-07-20T02:52:23.180880+0000 (oldest deadline 2023-07-20T02:52:50.781771+0000) 2023-07-20T02:53:07.203 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:06 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:06.842+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.782398+0000 front 2023-07-20T02:52:30.782308+0000 (oldest deadline 2023-07-20T02:52:54.282509+0000) 2023-07-20T02:53:07.203 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:06 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:06.842+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:36.583749+0000 front 2023-07-20T02:52:36.583925+0000 (oldest deadline 2023-07-20T02:53:01.883610+0000) 2023-07-20T02:53:07.392 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:07 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:07.160+0000 7f6a1ea83700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:53:07.392 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:07 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:07.160+0000 7f6a1ea83700 -1 osd.7 89 *** Got signal Terminated *** 2023-07-20T02:53:07.392 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:07 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:07.160+0000 7f6a1ea83700 -1 osd.7 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T02:53:07.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:07 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:07.116+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:07.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:07 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:07.116+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:07.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:07 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:07.116+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:07.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:07 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:07.116+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:38.634686+0000 front 2023-07-20T02:52:38.634805+0000 (oldest deadline 2023-07-20T02:53:03.934673+0000) 2023-07-20T02:53:07.449 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:07 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:07.008+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:53:07.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:07 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:07.008+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:53:07.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:07 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:07.008+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:29.391927+0000 front 2023-07-20T02:52:29.391909+0000 (oldest deadline 2023-07-20T02:52:55.292047+0000) 2023-07-20T02:53:07.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:07 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:07.008+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.393003+0000 front 2023-07-20T02:52:39.393069+0000 (oldest deadline 2023-07-20T02:53:02.193566+0000) 2023-07-20T02:53:07.464 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:07 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:07.189+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:07.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:07 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:07.189+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:07.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:07 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:07.189+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:08.214 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:07 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:07.879+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:53:08.215 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:07 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:07.879+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:23.180901+0000 front 2023-07-20T02:52:23.180880+0000 (oldest deadline 2023-07-20T02:52:50.781771+0000) 2023-07-20T02:53:08.215 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:07 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:07.879+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.782398+0000 front 2023-07-20T02:52:30.782308+0000 (oldest deadline 2023-07-20T02:52:54.282509+0000) 2023-07-20T02:53:08.215 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:07 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:07.879+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:36.583749+0000 front 2023-07-20T02:52:36.583925+0000 (oldest deadline 2023-07-20T02:53:01.883610+0000) 2023-07-20T02:53:08.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:08 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:08.092+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:08.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:08 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:08.092+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:08.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:08 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:08.092+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:08.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:08 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:08.092+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:38.634686+0000 front 2023-07-20T02:52:38.634805+0000 (oldest deadline 2023-07-20T02:53:03.934673+0000) 2023-07-20T02:53:08.393 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:07 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:07.901+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.563179+0000 front 2023-07-20T02:52:21.563231+0000 (oldest deadline 2023-07-20T02:52:47.463400+0000) 2023-07-20T02:53:08.393 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:07 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:07.901+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:32.164343+0000 front 2023-07-20T02:52:32.164405+0000 (oldest deadline 2023-07-20T02:52:54.464409+0000) 2023-07-20T02:53:08.393 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:07 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:07.901+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.765806+0000 front 2023-07-20T02:52:39.765630+0000 (oldest deadline 2023-07-20T02:53:01.465683+0000) 2023-07-20T02:53:08.449 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:08 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:08.043+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:53:08.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:08 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:08.043+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:53:08.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:08 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:08.043+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:29.391927+0000 front 2023-07-20T02:52:29.391909+0000 (oldest deadline 2023-07-20T02:52:55.292047+0000) 2023-07-20T02:53:08.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:08 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:08.043+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.393003+0000 front 2023-07-20T02:52:39.393069+0000 (oldest deadline 2023-07-20T02:53:02.193566+0000) 2023-07-20T02:53:08.714 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:08 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:08.216+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:08.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:08 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:08.216+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:08.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:08 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:08.216+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:09.214 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:08 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:08.889+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:53:09.215 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:08 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:08.889+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:23.180901+0000 front 2023-07-20T02:52:23.180880+0000 (oldest deadline 2023-07-20T02:52:50.781771+0000) 2023-07-20T02:53:09.215 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:08 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:08.889+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.782398+0000 front 2023-07-20T02:52:30.782308+0000 (oldest deadline 2023-07-20T02:52:54.282509+0000) 2023-07-20T02:53:09.215 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:08 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:08.889+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:36.583749+0000 front 2023-07-20T02:52:36.583925+0000 (oldest deadline 2023-07-20T02:53:01.883610+0000) 2023-07-20T02:53:09.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:09 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:09.053+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:09.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:09 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:09.053+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:09.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:09 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:09.053+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:09.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:09 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:09.053+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:38.634686+0000 front 2023-07-20T02:52:38.634805+0000 (oldest deadline 2023-07-20T02:53:03.934673+0000) 2023-07-20T02:53:09.393 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:08 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:08.927+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.563179+0000 front 2023-07-20T02:52:21.563231+0000 (oldest deadline 2023-07-20T02:52:47.463400+0000) 2023-07-20T02:53:09.393 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:08 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:08.927+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:32.164343+0000 front 2023-07-20T02:52:32.164405+0000 (oldest deadline 2023-07-20T02:52:54.464409+0000) 2023-07-20T02:53:09.393 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:08 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:08.928+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.765806+0000 front 2023-07-20T02:52:39.765630+0000 (oldest deadline 2023-07-20T02:53:01.465683+0000) 2023-07-20T02:53:09.449 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:09 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:09.053+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:53:09.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:09 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:09.053+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:53:09.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:09 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:09.053+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:29.391927+0000 front 2023-07-20T02:52:29.391909+0000 (oldest deadline 2023-07-20T02:52:55.292047+0000) 2023-07-20T02:53:09.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:09 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:09.053+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.393003+0000 front 2023-07-20T02:52:39.393069+0000 (oldest deadline 2023-07-20T02:53:02.193566+0000) 2023-07-20T02:53:09.714 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:09 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:09.219+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:09.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:09 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:09.219+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:09.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:09 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:09.219+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:10.211 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:09 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:09.895+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:53:10.212 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:09 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:09.895+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:23.180901+0000 front 2023-07-20T02:52:23.180880+0000 (oldest deadline 2023-07-20T02:52:50.781771+0000) 2023-07-20T02:53:10.212 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:09 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:09.895+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.782398+0000 front 2023-07-20T02:52:30.782308+0000 (oldest deadline 2023-07-20T02:52:54.282509+0000) 2023-07-20T02:53:10.212 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:09 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:09.895+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:36.583749+0000 front 2023-07-20T02:52:36.583925+0000 (oldest deadline 2023-07-20T02:53:01.883610+0000) 2023-07-20T02:53:10.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:10 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:10.014+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:10.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:10 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:10.014+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:10.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:10 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:10.014+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:10.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:10 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:10.014+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:38.634686+0000 front 2023-07-20T02:52:38.634805+0000 (oldest deadline 2023-07-20T02:53:03.934673+0000) 2023-07-20T02:53:10.393 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:09 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:09.977+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.563179+0000 front 2023-07-20T02:52:21.563231+0000 (oldest deadline 2023-07-20T02:52:47.463400+0000) 2023-07-20T02:53:10.393 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:09 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:09.977+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:32.164343+0000 front 2023-07-20T02:52:32.164405+0000 (oldest deadline 2023-07-20T02:52:54.464409+0000) 2023-07-20T02:53:10.393 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:09 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:09.977+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.765806+0000 front 2023-07-20T02:52:39.765630+0000 (oldest deadline 2023-07-20T02:53:01.465683+0000) 2023-07-20T02:53:10.449 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:10 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:10.028+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:53:10.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:10 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:10.028+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:53:10.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:10 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:10.028+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:29.391927+0000 front 2023-07-20T02:52:29.391909+0000 (oldest deadline 2023-07-20T02:52:55.292047+0000) 2023-07-20T02:53:10.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:10 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:10.028+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.393003+0000 front 2023-07-20T02:52:39.393069+0000 (oldest deadline 2023-07-20T02:53:02.193566+0000) 2023-07-20T02:53:10.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:10 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:10.197+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:10.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:10 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:10.197+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:10.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:10 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:10.197+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:11.175 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:10 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:10.883+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:53:11.188 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:10 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:10.883+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:23.180901+0000 front 2023-07-20T02:52:23.180880+0000 (oldest deadline 2023-07-20T02:52:50.781771+0000) 2023-07-20T02:53:11.188 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:10 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:10.883+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.782398+0000 front 2023-07-20T02:52:30.782308+0000 (oldest deadline 2023-07-20T02:52:54.282509+0000) 2023-07-20T02:53:11.188 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:10 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:10.883+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:36.583749+0000 front 2023-07-20T02:52:36.583925+0000 (oldest deadline 2023-07-20T02:53:01.883610+0000) 2023-07-20T02:53:11.391 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:10 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:10.976+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:11.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:10 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:10.976+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:11.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:10 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:10.976+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:11.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:10 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:10.976+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:38.634686+0000 front 2023-07-20T02:52:38.634805+0000 (oldest deadline 2023-07-20T02:53:03.934673+0000) 2023-07-20T02:53:11.392 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:11 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:11.016+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.563179+0000 front 2023-07-20T02:52:21.563231+0000 (oldest deadline 2023-07-20T02:52:47.463400+0000) 2023-07-20T02:53:11.393 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:11 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:11.016+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:32.164343+0000 front 2023-07-20T02:52:32.164405+0000 (oldest deadline 2023-07-20T02:52:54.464409+0000) 2023-07-20T02:53:11.393 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:11 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:11.016+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.765806+0000 front 2023-07-20T02:52:39.765630+0000 (oldest deadline 2023-07-20T02:53:01.465683+0000) 2023-07-20T02:53:11.449 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:10 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:10.988+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:53:11.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:10 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:10.988+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:53:11.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:10 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:10.988+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:29.391927+0000 front 2023-07-20T02:52:29.391909+0000 (oldest deadline 2023-07-20T02:52:55.292047+0000) 2023-07-20T02:53:11.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:10 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:10.988+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.393003+0000 front 2023-07-20T02:52:39.393069+0000 (oldest deadline 2023-07-20T02:53:02.193566+0000) 2023-07-20T02:53:11.464 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:11 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:11.161+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:11.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:11 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:11.161+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:11.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:11 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:11.161+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:12.188 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:11 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:11.884+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:53:12.189 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:11 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:11.884+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:23.180901+0000 front 2023-07-20T02:52:23.180880+0000 (oldest deadline 2023-07-20T02:52:50.781771+0000) 2023-07-20T02:53:12.189 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:11 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:11.884+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.782398+0000 front 2023-07-20T02:52:30.782308+0000 (oldest deadline 2023-07-20T02:52:54.282509+0000) 2023-07-20T02:53:12.189 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:11 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:11.884+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:36.583749+0000 front 2023-07-20T02:52:36.583925+0000 (oldest deadline 2023-07-20T02:53:01.883610+0000) 2023-07-20T02:53:12.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:11 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:11.931+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:12.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:11 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:11.931+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:12.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:11 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:11.931+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:12.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:11 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:11.931+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:38.634686+0000 front 2023-07-20T02:52:38.634805+0000 (oldest deadline 2023-07-20T02:53:03.934673+0000) 2023-07-20T02:53:12.393 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:11 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:11.998+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.563179+0000 front 2023-07-20T02:52:21.563231+0000 (oldest deadline 2023-07-20T02:52:47.463400+0000) 2023-07-20T02:53:12.393 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:11 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:11.998+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:32.164343+0000 front 2023-07-20T02:52:32.164405+0000 (oldest deadline 2023-07-20T02:52:54.464409+0000) 2023-07-20T02:53:12.393 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:11 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:11.998+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.765806+0000 front 2023-07-20T02:52:39.765630+0000 (oldest deadline 2023-07-20T02:53:01.465683+0000) 2023-07-20T02:53:12.393 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:11 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7[144905]: 2023-07-20T02:53:11.998+0000 7f6a17072700 -1 osd.7 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:46.067454+0000 front 2023-07-20T02:52:46.067501+0000 (oldest deadline 2023-07-20T02:53:11.967619+0000) 2023-07-20T02:53:12.449 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:12 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:12.016+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:53:12.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:12 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:12.016+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:53:12.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:12 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:12.016+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:29.391927+0000 front 2023-07-20T02:52:29.391909+0000 (oldest deadline 2023-07-20T02:52:55.292047+0000) 2023-07-20T02:53:12.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:12 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:12.016+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.393003+0000 front 2023-07-20T02:52:39.393069+0000 (oldest deadline 2023-07-20T02:53:02.193566+0000) 2023-07-20T02:53:12.464 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:12 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:12.174+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:12.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:12 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:12.174+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:12.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:12 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:12.174+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:12.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:12 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:12.174+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.906632+0000 front 2023-07-20T02:52:48.906737+0000 (oldest deadline 2023-07-20T02:53:11.806525+0000) 2023-07-20T02:53:12.892 INFO:journalctl@ceph.osd.7.smithi158.stdout:Jul 20 02:53:12 smithi158 bash[156389]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-7 2023-07-20T02:53:13.214 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:12 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:12.889+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:53:13.215 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:12 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:12.889+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:23.180901+0000 front 2023-07-20T02:52:23.180880+0000 (oldest deadline 2023-07-20T02:52:50.781771+0000) 2023-07-20T02:53:13.215 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:12 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:12.889+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.782398+0000 front 2023-07-20T02:52:30.782308+0000 (oldest deadline 2023-07-20T02:52:54.282509+0000) 2023-07-20T02:53:13.215 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:12 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:12.889+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:36.583749+0000 front 2023-07-20T02:52:36.583925+0000 (oldest deadline 2023-07-20T02:53:01.883610+0000) 2023-07-20T02:53:13.215 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:12 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:12.889+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.885433+0000 front 2023-07-20T02:52:48.885375+0000 (oldest deadline 2023-07-20T02:53:12.385374+0000) 2023-07-20T02:53:13.360 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:12 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:12.942+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:13.361 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:12 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:12.942+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:13.361 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:12 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:12.942+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:13.361 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:12 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:12.942+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:38.634686+0000 front 2023-07-20T02:52:38.634805+0000 (oldest deadline 2023-07-20T02:53:03.934673+0000) 2023-07-20T02:53:13.449 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:13 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:13.049+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:53:13.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:13 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:13.049+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:53:13.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:13 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:13.049+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:29.391927+0000 front 2023-07-20T02:52:29.391909+0000 (oldest deadline 2023-07-20T02:52:55.292047+0000) 2023-07-20T02:53:13.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:13 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:13.049+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.393003+0000 front 2023-07-20T02:52:39.393069+0000 (oldest deadline 2023-07-20T02:53:02.193566+0000) 2023-07-20T02:53:13.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:13 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:13.049+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:47.995711+0000 front 2023-07-20T02:52:47.995254+0000 (oldest deadline 2023-07-20T02:53:12.695300+0000) 2023-07-20T02:53:13.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:13 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:13.218+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:13.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:13 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:13.218+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:13.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:13 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:13.218+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:13.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:13 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:13.218+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.906632+0000 front 2023-07-20T02:52:48.906737+0000 (oldest deadline 2023-07-20T02:53:11.806525+0000) 2023-07-20T02:53:14.078 DEBUG:teuthology.orchestra.run.smithi158:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.7.service' 2023-07-20T02:53:14.119 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:53:14.120 INFO:tasks.cephadm.osd.7:Stopped osd.7 2023-07-20T02:53:14.120 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2023-07-20T02:53:14.121 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.8 2023-07-20T02:53:14.214 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:13 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:13.915+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:53:14.215 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:13 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:13.915+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:23.180901+0000 front 2023-07-20T02:52:23.180880+0000 (oldest deadline 2023-07-20T02:52:50.781771+0000) 2023-07-20T02:53:14.215 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:13 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:13.915+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.782398+0000 front 2023-07-20T02:52:30.782308+0000 (oldest deadline 2023-07-20T02:52:54.282509+0000) 2023-07-20T02:53:14.215 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:13 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:13.915+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:36.583749+0000 front 2023-07-20T02:52:36.583925+0000 (oldest deadline 2023-07-20T02:53:01.883610+0000) 2023-07-20T02:53:14.215 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:13 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:13.915+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.885433+0000 front 2023-07-20T02:52:48.885375+0000 (oldest deadline 2023-07-20T02:53:12.385374+0000) 2023-07-20T02:53:14.215 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:14 smithi188 systemd[1]: Stopping Ceph osd.8 for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:53:14.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:13 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:13.976+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:14.394 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:13 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:13.976+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:14.394 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:13 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:13.976+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:14.394 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:13 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:13.976+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:38.634686+0000 front 2023-07-20T02:52:38.634805+0000 (oldest deadline 2023-07-20T02:53:03.934673+0000) 2023-07-20T02:53:14.449 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:14 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:14.066+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:53:14.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:14 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:14.066+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:53:14.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:14 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:14.066+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:29.391927+0000 front 2023-07-20T02:52:29.391909+0000 (oldest deadline 2023-07-20T02:52:55.292047+0000) 2023-07-20T02:53:14.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:14 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:14.066+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.393003+0000 front 2023-07-20T02:52:39.393069+0000 (oldest deadline 2023-07-20T02:53:02.193566+0000) 2023-07-20T02:53:14.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:14 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:14.066+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:47.995711+0000 front 2023-07-20T02:52:47.995254+0000 (oldest deadline 2023-07-20T02:53:12.695300+0000) 2023-07-20T02:53:14.492 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:14 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:14.209+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:14.493 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:14 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:14.209+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:14.493 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:14 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:14.209+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:14.493 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:14 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:14.209+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.906632+0000 front 2023-07-20T02:52:48.906737+0000 (oldest deadline 2023-07-20T02:53:11.806525+0000) 2023-07-20T02:53:14.928 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:14 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:14.478+0000 7f608e4f6700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:53:14.928 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:14 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:14.478+0000 7f608e4f6700 -1 osd.8 89 *** Got signal Terminated *** 2023-07-20T02:53:14.928 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:14 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:14.478+0000 7f608e4f6700 -1 osd.8 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T02:53:14.928 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:14 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:14.914+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:53:15.215 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:14 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:14.914+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:23.180901+0000 front 2023-07-20T02:52:23.180880+0000 (oldest deadline 2023-07-20T02:52:50.781771+0000) 2023-07-20T02:53:15.215 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:14 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:14.914+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.782398+0000 front 2023-07-20T02:52:30.782308+0000 (oldest deadline 2023-07-20T02:52:54.282509+0000) 2023-07-20T02:53:15.215 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:14 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:14.914+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:36.583749+0000 front 2023-07-20T02:52:36.583925+0000 (oldest deadline 2023-07-20T02:53:01.883610+0000) 2023-07-20T02:53:15.216 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:14 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:14.914+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.885433+0000 front 2023-07-20T02:52:48.885375+0000 (oldest deadline 2023-07-20T02:53:12.385374+0000) 2023-07-20T02:53:15.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:15 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:15.012+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:15.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:15 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:15.012+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:15.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:15 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:15.012+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:15.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:15 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:15.012+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:38.634686+0000 front 2023-07-20T02:52:38.634805+0000 (oldest deadline 2023-07-20T02:53:03.934673+0000) 2023-07-20T02:53:15.449 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:15 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:15.093+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:53:15.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:15 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:15.093+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:53:15.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:15 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:15.093+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:29.391927+0000 front 2023-07-20T02:52:29.391909+0000 (oldest deadline 2023-07-20T02:52:55.292047+0000) 2023-07-20T02:53:15.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:15 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:15.093+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.393003+0000 front 2023-07-20T02:52:39.393069+0000 (oldest deadline 2023-07-20T02:53:02.193566+0000) 2023-07-20T02:53:15.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:15 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:15.093+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:47.995711+0000 front 2023-07-20T02:52:47.995254+0000 (oldest deadline 2023-07-20T02:53:12.695300+0000) 2023-07-20T02:53:15.714 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:15 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:15.239+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:15.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:15 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:15.239+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:15.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:15 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:15.239+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:15.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:15 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:15.239+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.906632+0000 front 2023-07-20T02:52:48.906737+0000 (oldest deadline 2023-07-20T02:53:11.806525+0000) 2023-07-20T02:53:16.215 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:15 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:15.943+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:53:16.215 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:15 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:15.943+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:23.180901+0000 front 2023-07-20T02:52:23.180880+0000 (oldest deadline 2023-07-20T02:52:50.781771+0000) 2023-07-20T02:53:16.215 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:15 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:15.943+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.782398+0000 front 2023-07-20T02:52:30.782308+0000 (oldest deadline 2023-07-20T02:52:54.282509+0000) 2023-07-20T02:53:16.216 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:15 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:15.943+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:36.583749+0000 front 2023-07-20T02:52:36.583925+0000 (oldest deadline 2023-07-20T02:53:01.883610+0000) 2023-07-20T02:53:16.216 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:15 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:15.943+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.885433+0000 front 2023-07-20T02:52:48.885375+0000 (oldest deadline 2023-07-20T02:53:12.385374+0000) 2023-07-20T02:53:16.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:16 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:16.010+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:16.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:16 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:16.010+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:16.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:16 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:16.010+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:16.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:16 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:16.010+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:38.634686+0000 front 2023-07-20T02:52:38.634805+0000 (oldest deadline 2023-07-20T02:53:03.934673+0000) 2023-07-20T02:53:16.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:16 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:16.010+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:43.934952+0000 front 2023-07-20T02:52:43.934968+0000 (oldest deadline 2023-07-20T02:53:15.135593+0000) 2023-07-20T02:53:16.449 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:16 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:16.125+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:53:16.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:16 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:16.125+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:53:16.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:16 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:16.125+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:29.391927+0000 front 2023-07-20T02:52:29.391909+0000 (oldest deadline 2023-07-20T02:52:55.292047+0000) 2023-07-20T02:53:16.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:16 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:16.125+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.393003+0000 front 2023-07-20T02:52:39.393069+0000 (oldest deadline 2023-07-20T02:53:02.193566+0000) 2023-07-20T02:53:16.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:16 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:16.125+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:47.995711+0000 front 2023-07-20T02:52:47.995254+0000 (oldest deadline 2023-07-20T02:53:12.695300+0000) 2023-07-20T02:53:16.714 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:16 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:16.259+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:16.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:16 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:16.259+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:16.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:16 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:16.259+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:16.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:16 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:16.259+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.906632+0000 front 2023-07-20T02:52:48.906737+0000 (oldest deadline 2023-07-20T02:53:11.806525+0000) 2023-07-20T02:53:17.254 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:16 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:16.977+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:53:17.254 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:16 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:16.977+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:23.180901+0000 front 2023-07-20T02:52:23.180880+0000 (oldest deadline 2023-07-20T02:52:50.781771+0000) 2023-07-20T02:53:17.254 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:16 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:16.977+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.782398+0000 front 2023-07-20T02:52:30.782308+0000 (oldest deadline 2023-07-20T02:52:54.282509+0000) 2023-07-20T02:53:17.255 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:16 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:16.977+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:36.583749+0000 front 2023-07-20T02:52:36.583925+0000 (oldest deadline 2023-07-20T02:53:01.883610+0000) 2023-07-20T02:53:17.255 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:16 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:16.977+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.885433+0000 front 2023-07-20T02:52:48.885375+0000 (oldest deadline 2023-07-20T02:53:12.385374+0000) 2023-07-20T02:53:17.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:16 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:16.987+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:17.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:16 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:16.987+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:17.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:16 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:16.987+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:17.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:16 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:16.987+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:38.634686+0000 front 2023-07-20T02:52:38.634805+0000 (oldest deadline 2023-07-20T02:53:03.934673+0000) 2023-07-20T02:53:17.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:16 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:16.987+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:43.934952+0000 front 2023-07-20T02:52:43.934968+0000 (oldest deadline 2023-07-20T02:53:15.135593+0000) 2023-07-20T02:53:17.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:16 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:16.987+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:55.136272+0000 front 2023-07-20T02:52:55.136208+0000 (oldest deadline 2023-07-20T02:53:16.836201+0000) 2023-07-20T02:53:17.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:17 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:17.160+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:53:17.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:17 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:17.160+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:53:17.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:17 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:17.160+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:29.391927+0000 front 2023-07-20T02:52:29.391909+0000 (oldest deadline 2023-07-20T02:52:55.292047+0000) 2023-07-20T02:53:17.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:17 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:17.160+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.393003+0000 front 2023-07-20T02:52:39.393069+0000 (oldest deadline 2023-07-20T02:53:02.193566+0000) 2023-07-20T02:53:17.451 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:17 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:17.160+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:47.995711+0000 front 2023-07-20T02:52:47.995254+0000 (oldest deadline 2023-07-20T02:53:12.695300+0000) 2023-07-20T02:53:17.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:17 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:17.240+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:17.716 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:17 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:17.240+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:17.716 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:17 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:17.240+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:17.716 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:17 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:17.240+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.906632+0000 front 2023-07-20T02:52:48.906737+0000 (oldest deadline 2023-07-20T02:53:11.806525+0000) 2023-07-20T02:53:18.276 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:17 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:17.967+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:53:18.276 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:17 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:17.967+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:23.180901+0000 front 2023-07-20T02:52:23.180880+0000 (oldest deadline 2023-07-20T02:52:50.781771+0000) 2023-07-20T02:53:18.276 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:17 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:17.967+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.782398+0000 front 2023-07-20T02:52:30.782308+0000 (oldest deadline 2023-07-20T02:52:54.282509+0000) 2023-07-20T02:53:18.277 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:17 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:17.967+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:36.583749+0000 front 2023-07-20T02:52:36.583925+0000 (oldest deadline 2023-07-20T02:53:01.883610+0000) 2023-07-20T02:53:18.277 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:17 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:17.967+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.885433+0000 front 2023-07-20T02:52:48.885375+0000 (oldest deadline 2023-07-20T02:53:12.385374+0000) 2023-07-20T02:53:18.277 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:17 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:17.967+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:55.886553+0000 front 2023-07-20T02:52:55.886577+0000 (oldest deadline 2023-07-20T02:53:17.586690+0000) 2023-07-20T02:53:18.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:17 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:17.952+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:18.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:17 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:17.952+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:18.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:17 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:17.952+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:18.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:17 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:17.952+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:38.634686+0000 front 2023-07-20T02:52:38.634805+0000 (oldest deadline 2023-07-20T02:53:03.934673+0000) 2023-07-20T02:53:18.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:17 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:17.952+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:43.934952+0000 front 2023-07-20T02:52:43.934968+0000 (oldest deadline 2023-07-20T02:53:15.135593+0000) 2023-07-20T02:53:18.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:17 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:17.952+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:55.136272+0000 front 2023-07-20T02:52:55.136208+0000 (oldest deadline 2023-07-20T02:53:16.836201+0000) 2023-07-20T02:53:18.451 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:18 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:18.111+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:53:18.452 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:18 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:18.111+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:53:18.452 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:18 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:18.111+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:29.391927+0000 front 2023-07-20T02:52:29.391909+0000 (oldest deadline 2023-07-20T02:52:55.292047+0000) 2023-07-20T02:53:18.452 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:18 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:18.111+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.393003+0000 front 2023-07-20T02:52:39.393069+0000 (oldest deadline 2023-07-20T02:53:02.193566+0000) 2023-07-20T02:53:18.452 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:18 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:18.111+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:47.995711+0000 front 2023-07-20T02:52:47.995254+0000 (oldest deadline 2023-07-20T02:53:12.695300+0000) 2023-07-20T02:53:18.453 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:18 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:18.111+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:53.196093+0000 front 2023-07-20T02:52:53.196310+0000 (oldest deadline 2023-07-20T02:53:17.296264+0000) 2023-07-20T02:53:18.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:18 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:18.262+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:18.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:18 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:18.262+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:18.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:18 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:18.262+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:18.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:18 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:18.262+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.906632+0000 front 2023-07-20T02:52:48.906737+0000 (oldest deadline 2023-07-20T02:53:11.806525+0000) 2023-07-20T02:53:19.215 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:18 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:18.947+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.079198+0000 front 2023-07-20T02:52:15.079211+0000 (oldest deadline 2023-07-20T02:52:39.179228+0000) 2023-07-20T02:53:19.215 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:18 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:18.947+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:23.180901+0000 front 2023-07-20T02:52:23.180880+0000 (oldest deadline 2023-07-20T02:52:50.781771+0000) 2023-07-20T02:53:19.215 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:18 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:18.947+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.782398+0000 front 2023-07-20T02:52:30.782308+0000 (oldest deadline 2023-07-20T02:52:54.282509+0000) 2023-07-20T02:53:19.215 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:18 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:18.947+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:36.583749+0000 front 2023-07-20T02:52:36.583925+0000 (oldest deadline 2023-07-20T02:53:01.883610+0000) 2023-07-20T02:53:19.215 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:18 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:18.947+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.885433+0000 front 2023-07-20T02:52:48.885375+0000 (oldest deadline 2023-07-20T02:53:12.385374+0000) 2023-07-20T02:53:19.215 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:18 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8[146426]: 2023-07-20T02:53:18.947+0000 7f6087ae7700 -1 osd.8 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:55.886553+0000 front 2023-07-20T02:52:55.886577+0000 (oldest deadline 2023-07-20T02:53:17.586690+0000) 2023-07-20T02:53:19.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:18 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:18.968+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:19.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:18 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:18.968+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:19.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:18 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:18.968+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:19.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:18 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:18.968+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:38.634686+0000 front 2023-07-20T02:52:38.634805+0000 (oldest deadline 2023-07-20T02:53:03.934673+0000) 2023-07-20T02:53:19.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:18 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:18.968+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:43.934952+0000 front 2023-07-20T02:52:43.934968+0000 (oldest deadline 2023-07-20T02:53:15.135593+0000) 2023-07-20T02:53:19.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:18 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:18.968+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:55.136272+0000 front 2023-07-20T02:52:55.136208+0000 (oldest deadline 2023-07-20T02:53:16.836201+0000) 2023-07-20T02:53:19.449 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:19 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:19.065+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:53:19.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:19 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:19.065+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:53:19.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:19 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:19.065+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:29.391927+0000 front 2023-07-20T02:52:29.391909+0000 (oldest deadline 2023-07-20T02:52:55.292047+0000) 2023-07-20T02:53:19.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:19 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:19.065+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.393003+0000 front 2023-07-20T02:52:39.393069+0000 (oldest deadline 2023-07-20T02:53:02.193566+0000) 2023-07-20T02:53:19.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:19 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:19.065+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:47.995711+0000 front 2023-07-20T02:52:47.995254+0000 (oldest deadline 2023-07-20T02:53:12.695300+0000) 2023-07-20T02:53:19.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:19 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:19.065+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:53.196093+0000 front 2023-07-20T02:52:53.196310+0000 (oldest deadline 2023-07-20T02:53:17.296264+0000) 2023-07-20T02:53:19.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:19 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:19.229+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:19.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:19 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:19.229+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:19.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:19 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:19.229+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:19.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:19 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:19.229+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.906632+0000 front 2023-07-20T02:52:48.906737+0000 (oldest deadline 2023-07-20T02:53:11.806525+0000) 2023-07-20T02:53:20.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:19 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:19.988+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:20.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:19 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:19.988+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:20.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:19 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:19.988+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:20.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:19 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:19.988+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:38.634686+0000 front 2023-07-20T02:52:38.634805+0000 (oldest deadline 2023-07-20T02:53:03.934673+0000) 2023-07-20T02:53:20.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:19 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:19.988+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:43.934952+0000 front 2023-07-20T02:52:43.934968+0000 (oldest deadline 2023-07-20T02:53:15.135593+0000) 2023-07-20T02:53:20.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:19 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:19.988+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:55.136272+0000 front 2023-07-20T02:52:55.136208+0000 (oldest deadline 2023-07-20T02:53:16.836201+0000) 2023-07-20T02:53:20.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:20 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:20.070+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:53:20.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:20 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:20.070+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:53:20.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:20 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:20.070+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:29.391927+0000 front 2023-07-20T02:52:29.391909+0000 (oldest deadline 2023-07-20T02:52:55.292047+0000) 2023-07-20T02:53:20.451 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:20 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:20.070+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.393003+0000 front 2023-07-20T02:52:39.393069+0000 (oldest deadline 2023-07-20T02:53:02.193566+0000) 2023-07-20T02:53:20.451 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:20 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:20.070+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:47.995711+0000 front 2023-07-20T02:52:47.995254+0000 (oldest deadline 2023-07-20T02:53:12.695300+0000) 2023-07-20T02:53:20.451 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:20 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:20.070+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:53.196093+0000 front 2023-07-20T02:52:53.196310+0000 (oldest deadline 2023-07-20T02:53:17.296264+0000) 2023-07-20T02:53:20.465 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 20 02:53:20 smithi188 bash[158136]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-8 2023-07-20T02:53:20.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:20 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:20.203+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:20.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:20 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:20.203+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:20.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:20 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:20.204+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:20.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:20 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:20.204+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.906632+0000 front 2023-07-20T02:52:48.906737+0000 (oldest deadline 2023-07-20T02:53:11.806525+0000) 2023-07-20T02:53:21.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:20 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:20.990+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:21.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:20 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:20.990+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:21.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:20 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:20.990+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:21.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:20 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:20.990+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:38.634686+0000 front 2023-07-20T02:52:38.634805+0000 (oldest deadline 2023-07-20T02:53:03.934673+0000) 2023-07-20T02:53:21.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:20 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:20.990+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:43.934952+0000 front 2023-07-20T02:52:43.934968+0000 (oldest deadline 2023-07-20T02:53:15.135593+0000) 2023-07-20T02:53:21.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:20 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:20.990+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:55.136272+0000 front 2023-07-20T02:52:55.136208+0000 (oldest deadline 2023-07-20T02:53:16.836201+0000) 2023-07-20T02:53:21.452 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:21 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:21.078+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:53:21.452 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:21 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:21.078+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:53:21.452 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:21 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:21.078+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:29.391927+0000 front 2023-07-20T02:52:29.391909+0000 (oldest deadline 2023-07-20T02:52:55.292047+0000) 2023-07-20T02:53:21.452 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:21 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:21.078+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.393003+0000 front 2023-07-20T02:52:39.393069+0000 (oldest deadline 2023-07-20T02:53:02.193566+0000) 2023-07-20T02:53:21.453 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:21 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:21.078+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:47.995711+0000 front 2023-07-20T02:52:47.995254+0000 (oldest deadline 2023-07-20T02:53:12.695300+0000) 2023-07-20T02:53:21.453 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:21 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:21.078+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:53.196093+0000 front 2023-07-20T02:52:53.196310+0000 (oldest deadline 2023-07-20T02:53:17.296264+0000) 2023-07-20T02:53:21.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:21 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:21.195+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:21.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:21 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:21.195+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:21.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:21 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:21.195+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:21.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:21 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:21.195+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.906632+0000 front 2023-07-20T02:52:48.906737+0000 (oldest deadline 2023-07-20T02:53:11.806525+0000) 2023-07-20T02:53:21.848 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.8.service' 2023-07-20T02:53:21.889 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:53:21.891 INFO:tasks.cephadm.osd.8:Stopped osd.8 2023-07-20T02:53:21.892 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2023-07-20T02:53:21.893 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.9 2023-07-20T02:53:22.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:21 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:21.992+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:22.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:21 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:21.992+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:22.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:21 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:21.992+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:22.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:21 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:21.992+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:38.634686+0000 front 2023-07-20T02:52:38.634805+0000 (oldest deadline 2023-07-20T02:53:03.934673+0000) 2023-07-20T02:53:22.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:21 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:21.992+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:43.934952+0000 front 2023-07-20T02:52:43.934968+0000 (oldest deadline 2023-07-20T02:53:15.135593+0000) 2023-07-20T02:53:22.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:21 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:21.992+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:55.136272+0000 front 2023-07-20T02:52:55.136208+0000 (oldest deadline 2023-07-20T02:53:16.836201+0000) 2023-07-20T02:53:22.428 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:21 smithi007 systemd[1]: Stopping Ceph osd.9 for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:53:22.429 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:22 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:22.116+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:53:22.429 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:22 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:22.116+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:53:22.429 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:22 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:22.116+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:29.391927+0000 front 2023-07-20T02:52:29.391909+0000 (oldest deadline 2023-07-20T02:52:55.292047+0000) 2023-07-20T02:53:22.429 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:22 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:22.116+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.393003+0000 front 2023-07-20T02:52:39.393069+0000 (oldest deadline 2023-07-20T02:53:02.193566+0000) 2023-07-20T02:53:22.429 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:22 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:22.116+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:47.995711+0000 front 2023-07-20T02:52:47.995254+0000 (oldest deadline 2023-07-20T02:53:12.695300+0000) 2023-07-20T02:53:22.429 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:22 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:22.116+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:53.196093+0000 front 2023-07-20T02:52:53.196310+0000 (oldest deadline 2023-07-20T02:53:17.296264+0000) 2023-07-20T02:53:22.429 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:22 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:22.413+0000 7f8379574700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:53:22.464 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:22 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:22.152+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:22.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:22 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:22.152+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:22.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:22 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:22.152+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:22.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:22 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:22.152+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.906632+0000 front 2023-07-20T02:52:48.906737+0000 (oldest deadline 2023-07-20T02:53:11.806525+0000) 2023-07-20T02:53:22.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:22 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:22.152+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:51.806912+0000 front 2023-07-20T02:52:51.806892+0000 (oldest deadline 2023-07-20T02:53:21.207783+0000) 2023-07-20T02:53:22.699 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:22 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:22.413+0000 7f8379574700 -1 osd.9 89 *** Got signal Terminated *** 2023-07-20T02:53:22.700 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:22 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:22.413+0000 7f8379574700 -1 osd.9 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T02:53:23.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:23 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:23.013+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:23.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:23 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:23.013+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:23.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:23 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:23.013+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:23.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:23 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:23.013+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:38.634686+0000 front 2023-07-20T02:52:38.634805+0000 (oldest deadline 2023-07-20T02:53:03.934673+0000) 2023-07-20T02:53:23.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:23 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:23.013+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:43.934952+0000 front 2023-07-20T02:52:43.934968+0000 (oldest deadline 2023-07-20T02:53:15.135593+0000) 2023-07-20T02:53:23.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:23 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:23.013+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:55.136272+0000 front 2023-07-20T02:52:55.136208+0000 (oldest deadline 2023-07-20T02:53:16.836201+0000) 2023-07-20T02:53:23.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:23 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:23.144+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:53:23.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:23 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:23.144+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:53:23.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:23 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:23.144+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:29.391927+0000 front 2023-07-20T02:52:29.391909+0000 (oldest deadline 2023-07-20T02:52:55.292047+0000) 2023-07-20T02:53:23.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:23 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:23.144+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.393003+0000 front 2023-07-20T02:52:39.393069+0000 (oldest deadline 2023-07-20T02:53:02.193566+0000) 2023-07-20T02:53:23.451 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:23 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:23.144+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:47.995711+0000 front 2023-07-20T02:52:47.995254+0000 (oldest deadline 2023-07-20T02:53:12.695300+0000) 2023-07-20T02:53:23.451 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:23 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:23.144+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:53.196093+0000 front 2023-07-20T02:52:53.196310+0000 (oldest deadline 2023-07-20T02:53:17.296264+0000) 2023-07-20T02:53:23.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:23 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:23.161+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:23.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:23 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:23.161+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:23.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:23 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:23.161+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:23.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:23 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:23.162+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.906632+0000 front 2023-07-20T02:52:48.906737+0000 (oldest deadline 2023-07-20T02:53:11.806525+0000) 2023-07-20T02:53:23.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:23 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:23.162+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:51.806912+0000 front 2023-07-20T02:52:51.806892+0000 (oldest deadline 2023-07-20T02:53:21.207783+0000) 2023-07-20T02:53:24.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:23 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:23.976+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:24.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:23 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:23.976+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:24.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:23 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:23.976+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:24.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:23 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:23.976+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:38.634686+0000 front 2023-07-20T02:52:38.634805+0000 (oldest deadline 2023-07-20T02:53:03.934673+0000) 2023-07-20T02:53:24.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:23 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:23.976+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:43.934952+0000 front 2023-07-20T02:52:43.934968+0000 (oldest deadline 2023-07-20T02:53:15.135593+0000) 2023-07-20T02:53:24.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:23 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:23.976+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:55.136272+0000 front 2023-07-20T02:52:55.136208+0000 (oldest deadline 2023-07-20T02:53:16.836201+0000) 2023-07-20T02:53:24.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:24 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:24.152+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:53:24.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:24 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:24.152+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:53:24.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:24 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:24.152+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:29.391927+0000 front 2023-07-20T02:52:29.391909+0000 (oldest deadline 2023-07-20T02:52:55.292047+0000) 2023-07-20T02:53:24.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:24 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:24.152+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.393003+0000 front 2023-07-20T02:52:39.393069+0000 (oldest deadline 2023-07-20T02:53:02.193566+0000) 2023-07-20T02:53:24.451 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:24 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:24.152+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:47.995711+0000 front 2023-07-20T02:52:47.995254+0000 (oldest deadline 2023-07-20T02:53:12.695300+0000) 2023-07-20T02:53:24.451 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:24 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:24.152+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:53.196093+0000 front 2023-07-20T02:52:53.196310+0000 (oldest deadline 2023-07-20T02:53:17.296264+0000) 2023-07-20T02:53:24.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:24 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:24.211+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:24.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:24 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:24.211+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:24.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:24 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:24.211+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:24.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:24 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:24.211+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.906632+0000 front 2023-07-20T02:52:48.906737+0000 (oldest deadline 2023-07-20T02:53:11.806525+0000) 2023-07-20T02:53:24.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:24 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:24.211+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:51.806912+0000 front 2023-07-20T02:52:51.806892+0000 (oldest deadline 2023-07-20T02:53:21.207783+0000) 2023-07-20T02:53:25.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:25 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:25.016+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:25.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:25 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:25.016+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:25.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:25 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:25.016+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:25.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:25 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:25.016+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:38.634686+0000 front 2023-07-20T02:52:38.634805+0000 (oldest deadline 2023-07-20T02:53:03.934673+0000) 2023-07-20T02:53:25.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:25 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:25.016+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:43.934952+0000 front 2023-07-20T02:52:43.934968+0000 (oldest deadline 2023-07-20T02:53:15.135593+0000) 2023-07-20T02:53:25.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:25 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:25.016+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:55.136272+0000 front 2023-07-20T02:52:55.136208+0000 (oldest deadline 2023-07-20T02:53:16.836201+0000) 2023-07-20T02:53:25.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:25 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:25.153+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:53:25.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:25 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:25.153+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:53:25.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:25 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:25.153+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:29.391927+0000 front 2023-07-20T02:52:29.391909+0000 (oldest deadline 2023-07-20T02:52:55.292047+0000) 2023-07-20T02:53:25.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:25 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:25.153+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.393003+0000 front 2023-07-20T02:52:39.393069+0000 (oldest deadline 2023-07-20T02:53:02.193566+0000) 2023-07-20T02:53:25.451 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:25 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:25.153+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:47.995711+0000 front 2023-07-20T02:52:47.995254+0000 (oldest deadline 2023-07-20T02:53:12.695300+0000) 2023-07-20T02:53:25.451 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:25 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:25.153+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:53.196093+0000 front 2023-07-20T02:52:53.196310+0000 (oldest deadline 2023-07-20T02:53:17.296264+0000) 2023-07-20T02:53:25.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:25 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:25.214+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:25.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:25 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:25.214+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:25.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:25 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:25.214+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:25.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:25 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:25.214+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.906632+0000 front 2023-07-20T02:52:48.906737+0000 (oldest deadline 2023-07-20T02:53:11.806525+0000) 2023-07-20T02:53:25.716 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:25 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:25.214+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:51.806912+0000 front 2023-07-20T02:52:51.806892+0000 (oldest deadline 2023-07-20T02:53:21.207783+0000) 2023-07-20T02:53:25.716 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:25 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:25.214+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2023-07-20T02:53:01.208430+0000 front 2023-07-20T02:53:01.208470+0000 (oldest deadline 2023-07-20T02:53:24.708119+0000) 2023-07-20T02:53:26.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:25 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:25.975+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:26.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:25 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:25.975+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:26.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:25 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:25.975+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:26.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:25 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:25.975+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:38.634686+0000 front 2023-07-20T02:52:38.634805+0000 (oldest deadline 2023-07-20T02:53:03.934673+0000) 2023-07-20T02:53:26.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:25 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:25.975+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:43.934952+0000 front 2023-07-20T02:52:43.934968+0000 (oldest deadline 2023-07-20T02:53:15.135593+0000) 2023-07-20T02:53:26.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:25 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:25.975+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:55.136272+0000 front 2023-07-20T02:52:55.136208+0000 (oldest deadline 2023-07-20T02:53:16.836201+0000) 2023-07-20T02:53:26.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:26 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:26.155+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:53:26.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:26 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:26.155+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:53:26.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:26 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:26.155+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:29.391927+0000 front 2023-07-20T02:52:29.391909+0000 (oldest deadline 2023-07-20T02:52:55.292047+0000) 2023-07-20T02:53:26.451 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:26 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:26.155+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.393003+0000 front 2023-07-20T02:52:39.393069+0000 (oldest deadline 2023-07-20T02:53:02.193566+0000) 2023-07-20T02:53:26.451 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:26 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:26.155+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:47.995711+0000 front 2023-07-20T02:52:47.995254+0000 (oldest deadline 2023-07-20T02:53:12.695300+0000) 2023-07-20T02:53:26.451 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:26 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:26.155+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:53.196093+0000 front 2023-07-20T02:52:53.196310+0000 (oldest deadline 2023-07-20T02:53:17.296264+0000) 2023-07-20T02:53:26.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:26 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:26.210+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:26.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:26 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:26.210+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:26.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:26 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:26.210+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:26.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:26 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:26.210+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.906632+0000 front 2023-07-20T02:52:48.906737+0000 (oldest deadline 2023-07-20T02:53:11.806525+0000) 2023-07-20T02:53:26.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:26 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:26.210+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:51.806912+0000 front 2023-07-20T02:52:51.806892+0000 (oldest deadline 2023-07-20T02:53:21.207783+0000) 2023-07-20T02:53:26.716 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:26 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:26.210+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2023-07-20T02:53:01.208430+0000 front 2023-07-20T02:53:01.208470+0000 (oldest deadline 2023-07-20T02:53:24.708119+0000) 2023-07-20T02:53:27.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:27 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:27.017+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:27.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:27 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:27.017+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:27.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:27 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:27.017+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:27.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:27 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:27.017+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:38.634686+0000 front 2023-07-20T02:52:38.634805+0000 (oldest deadline 2023-07-20T02:53:03.934673+0000) 2023-07-20T02:53:27.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:27 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:27.017+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:43.934952+0000 front 2023-07-20T02:52:43.934968+0000 (oldest deadline 2023-07-20T02:53:15.135593+0000) 2023-07-20T02:53:27.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:27 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:27.017+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:55.136272+0000 front 2023-07-20T02:52:55.136208+0000 (oldest deadline 2023-07-20T02:53:16.836201+0000) 2023-07-20T02:53:27.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:27 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:27.114+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:16.088242+0000 front 2023-07-20T02:52:16.088279+0000 (oldest deadline 2023-07-20T02:52:39.588336+0000) 2023-07-20T02:53:27.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:27 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:27.114+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:22.990759+0000 front 2023-07-20T02:52:22.990787+0000 (oldest deadline 2023-07-20T02:52:46.490727+0000) 2023-07-20T02:53:27.450 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:27 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:27.114+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:29.391927+0000 front 2023-07-20T02:52:29.391909+0000 (oldest deadline 2023-07-20T02:52:55.292047+0000) 2023-07-20T02:53:27.451 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:27 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:27.114+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:39.393003+0000 front 2023-07-20T02:52:39.393069+0000 (oldest deadline 2023-07-20T02:53:02.193566+0000) 2023-07-20T02:53:27.451 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:27 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:27.114+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:47.995711+0000 front 2023-07-20T02:52:47.995254+0000 (oldest deadline 2023-07-20T02:53:12.695300+0000) 2023-07-20T02:53:27.451 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:27 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9[155944]: 2023-07-20T02:53:27.114+0000 7f8373b67700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:53.196093+0000 front 2023-07-20T02:52:53.196310+0000 (oldest deadline 2023-07-20T02:53:17.296264+0000) 2023-07-20T02:53:27.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:27 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:27.170+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:27.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:27 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:27.170+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:27.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:27 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:27.170+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:27.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:27 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:27.170+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.906632+0000 front 2023-07-20T02:52:48.906737+0000 (oldest deadline 2023-07-20T02:53:11.806525+0000) 2023-07-20T02:53:27.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:27 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:27.170+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:51.806912+0000 front 2023-07-20T02:52:51.806892+0000 (oldest deadline 2023-07-20T02:53:21.207783+0000) 2023-07-20T02:53:27.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:27 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:27.170+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2023-07-20T02:53:01.208430+0000 front 2023-07-20T02:53:01.208470+0000 (oldest deadline 2023-07-20T02:53:24.708119+0000) 2023-07-20T02:53:28.200 INFO:journalctl@ceph.osd.9.smithi007.stdout:Jul 20 02:53:27 smithi007 bash[166756]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-9 2023-07-20T02:53:28.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:27 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:27.974+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:28.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:27 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:27.974+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:28.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:27 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:27.974+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:28.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:27 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:27.974+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:38.634686+0000 front 2023-07-20T02:52:38.634805+0000 (oldest deadline 2023-07-20T02:53:03.934673+0000) 2023-07-20T02:53:28.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:27 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:27.974+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:43.934952+0000 front 2023-07-20T02:52:43.934968+0000 (oldest deadline 2023-07-20T02:53:15.135593+0000) 2023-07-20T02:53:28.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:27 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:27.974+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:55.136272+0000 front 2023-07-20T02:52:55.136208+0000 (oldest deadline 2023-07-20T02:53:16.836201+0000) 2023-07-20T02:53:28.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:28 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:28.153+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:28.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:28 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:28.153+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:28.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:28 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:28.153+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:28.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:28 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:28.153+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.906632+0000 front 2023-07-20T02:52:48.906737+0000 (oldest deadline 2023-07-20T02:53:11.806525+0000) 2023-07-20T02:53:28.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:28 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:28.153+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:51.806912+0000 front 2023-07-20T02:52:51.806892+0000 (oldest deadline 2023-07-20T02:53:21.207783+0000) 2023-07-20T02:53:28.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:28 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:28.153+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2023-07-20T02:53:01.208430+0000 front 2023-07-20T02:53:01.208470+0000 (oldest deadline 2023-07-20T02:53:24.708119+0000) 2023-07-20T02:53:29.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:29 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:29.002+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:29.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:29 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:29.002+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:29.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:29 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:29.002+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:29.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:29 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:29.002+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:38.634686+0000 front 2023-07-20T02:52:38.634805+0000 (oldest deadline 2023-07-20T02:53:03.934673+0000) 2023-07-20T02:53:29.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:29 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:29.002+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:43.934952+0000 front 2023-07-20T02:52:43.934968+0000 (oldest deadline 2023-07-20T02:53:15.135593+0000) 2023-07-20T02:53:29.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:29 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:29.002+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:55.136272+0000 front 2023-07-20T02:52:55.136208+0000 (oldest deadline 2023-07-20T02:53:16.836201+0000) 2023-07-20T02:53:29.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:29 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:29.002+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2023-07-20T02:53:02.737385+0000 front 2023-07-20T02:53:02.737273+0000 (oldest deadline 2023-07-20T02:53:28.637249+0000) 2023-07-20T02:53:29.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:29 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:29.188+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:29.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:29 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:29.188+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:29.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:29 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:29.188+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:29.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:29 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:29.188+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.906632+0000 front 2023-07-20T02:52:48.906737+0000 (oldest deadline 2023-07-20T02:53:11.806525+0000) 2023-07-20T02:53:29.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:29 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:29.188+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:51.806912+0000 front 2023-07-20T02:52:51.806892+0000 (oldest deadline 2023-07-20T02:53:21.207783+0000) 2023-07-20T02:53:29.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:29 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:29.188+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2023-07-20T02:53:01.208430+0000 front 2023-07-20T02:53:01.208470+0000 (oldest deadline 2023-07-20T02:53:24.708119+0000) 2023-07-20T02:53:29.813 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.9.service' 2023-07-20T02:53:29.906 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:53:29.907 INFO:tasks.cephadm.osd.9:Stopped osd.9 2023-07-20T02:53:29.908 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2023-07-20T02:53:29.909 DEBUG:teuthology.orchestra.run.smithi158:> sudo systemctl stop ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.10 2023-07-20T02:53:30.280 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:29 smithi158 systemd[1]: Stopping Ceph osd.10 for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:53:30.280 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:29 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:29.996+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:30.280 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:29 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:29.996+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:30.280 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:29 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:29.996+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:30.281 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:29 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:29.996+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:38.634686+0000 front 2023-07-20T02:52:38.634805+0000 (oldest deadline 2023-07-20T02:53:03.934673+0000) 2023-07-20T02:53:30.281 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:29 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:29.996+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:43.934952+0000 front 2023-07-20T02:52:43.934968+0000 (oldest deadline 2023-07-20T02:53:15.135593+0000) 2023-07-20T02:53:30.281 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:29 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:29.996+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:55.136272+0000 front 2023-07-20T02:52:55.136208+0000 (oldest deadline 2023-07-20T02:53:16.836201+0000) 2023-07-20T02:53:30.281 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:29 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:29.996+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2023-07-20T02:53:02.737385+0000 front 2023-07-20T02:53:02.737273+0000 (oldest deadline 2023-07-20T02:53:28.637249+0000) 2023-07-20T02:53:30.642 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:30 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:30.265+0000 7fddb89aa700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:53:30.642 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:30 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:30.265+0000 7fddb89aa700 -1 osd.10 89 *** Got signal Terminated *** 2023-07-20T02:53:30.642 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:30 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:30.265+0000 7fddb89aa700 -1 osd.10 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T02:53:30.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:30 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:30.231+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:30.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:30 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:30.231+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:30.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:30 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:30.231+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:30.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:30 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:30.231+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.906632+0000 front 2023-07-20T02:52:48.906737+0000 (oldest deadline 2023-07-20T02:53:11.806525+0000) 2023-07-20T02:53:30.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:30 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:30.231+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:51.806912+0000 front 2023-07-20T02:52:51.806892+0000 (oldest deadline 2023-07-20T02:53:21.207783+0000) 2023-07-20T02:53:30.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:30 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:30.231+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2023-07-20T02:53:01.208430+0000 front 2023-07-20T02:53:01.208470+0000 (oldest deadline 2023-07-20T02:53:24.708119+0000) 2023-07-20T02:53:31.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:31 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:31.037+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:31.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:31 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:31.037+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:31.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:31 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:31.037+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:31.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:31 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:31.037+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:38.634686+0000 front 2023-07-20T02:52:38.634805+0000 (oldest deadline 2023-07-20T02:53:03.934673+0000) 2023-07-20T02:53:31.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:31 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:31.037+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:43.934952+0000 front 2023-07-20T02:52:43.934968+0000 (oldest deadline 2023-07-20T02:53:15.135593+0000) 2023-07-20T02:53:31.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:31 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:31.037+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:55.136272+0000 front 2023-07-20T02:52:55.136208+0000 (oldest deadline 2023-07-20T02:53:16.836201+0000) 2023-07-20T02:53:31.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:31 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:31.037+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2023-07-20T02:53:02.737385+0000 front 2023-07-20T02:53:02.737273+0000 (oldest deadline 2023-07-20T02:53:28.637249+0000) 2023-07-20T02:53:31.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:31 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:31.273+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:31.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:31 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:31.273+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:31.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:31 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:31.273+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:31.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:31 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:31.273+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.906632+0000 front 2023-07-20T02:52:48.906737+0000 (oldest deadline 2023-07-20T02:53:11.806525+0000) 2023-07-20T02:53:31.716 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:31 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:31.273+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:51.806912+0000 front 2023-07-20T02:52:51.806892+0000 (oldest deadline 2023-07-20T02:53:21.207783+0000) 2023-07-20T02:53:31.716 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:31 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:31.273+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2023-07-20T02:53:01.208430+0000 front 2023-07-20T02:53:01.208470+0000 (oldest deadline 2023-07-20T02:53:24.708119+0000) 2023-07-20T02:53:32.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:32 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:32.029+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:32.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:32 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:32.029+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:32.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:32 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:32.029+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:32.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:32 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:32.029+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:38.634686+0000 front 2023-07-20T02:52:38.634805+0000 (oldest deadline 2023-07-20T02:53:03.934673+0000) 2023-07-20T02:53:32.394 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:32 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:32.029+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:43.934952+0000 front 2023-07-20T02:52:43.934968+0000 (oldest deadline 2023-07-20T02:53:15.135593+0000) 2023-07-20T02:53:32.394 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:32 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:32.029+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:55.136272+0000 front 2023-07-20T02:52:55.136208+0000 (oldest deadline 2023-07-20T02:53:16.836201+0000) 2023-07-20T02:53:32.395 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:32 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:32.029+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2023-07-20T02:53:02.737385+0000 front 2023-07-20T02:53:02.737273+0000 (oldest deadline 2023-07-20T02:53:28.637249+0000) 2023-07-20T02:53:32.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:32 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:32.241+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:32.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:32 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:32.241+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:32.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:32 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:32.241+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:32.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:32 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:32.241+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.906632+0000 front 2023-07-20T02:52:48.906737+0000 (oldest deadline 2023-07-20T02:53:11.806525+0000) 2023-07-20T02:53:32.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:32 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:32.241+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:51.806912+0000 front 2023-07-20T02:52:51.806892+0000 (oldest deadline 2023-07-20T02:53:21.207783+0000) 2023-07-20T02:53:32.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:32 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:32.241+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2023-07-20T02:53:01.208430+0000 front 2023-07-20T02:53:01.208470+0000 (oldest deadline 2023-07-20T02:53:24.708119+0000) 2023-07-20T02:53:33.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:33 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:33.001+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:33.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:33 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:33.001+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:33.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:33 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:33.001+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:33.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:33 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:33.001+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:38.634686+0000 front 2023-07-20T02:52:38.634805+0000 (oldest deadline 2023-07-20T02:53:03.934673+0000) 2023-07-20T02:53:33.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:33 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:33.001+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:43.934952+0000 front 2023-07-20T02:52:43.934968+0000 (oldest deadline 2023-07-20T02:53:15.135593+0000) 2023-07-20T02:53:33.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:33 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:33.001+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:55.136272+0000 front 2023-07-20T02:52:55.136208+0000 (oldest deadline 2023-07-20T02:53:16.836201+0000) 2023-07-20T02:53:33.394 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:33 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:33.001+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2023-07-20T02:53:02.737385+0000 front 2023-07-20T02:53:02.737273+0000 (oldest deadline 2023-07-20T02:53:28.637249+0000) 2023-07-20T02:53:33.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:33 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:33.219+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:33.715 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:33 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:33.219+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:33.716 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:33 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:33.219+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:33.716 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:33 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:33.219+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.906632+0000 front 2023-07-20T02:52:48.906737+0000 (oldest deadline 2023-07-20T02:53:11.806525+0000) 2023-07-20T02:53:33.716 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:33 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:33.219+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:51.806912+0000 front 2023-07-20T02:52:51.806892+0000 (oldest deadline 2023-07-20T02:53:21.207783+0000) 2023-07-20T02:53:33.717 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:33 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:33.219+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2023-07-20T02:53:01.208430+0000 front 2023-07-20T02:53:01.208470+0000 (oldest deadline 2023-07-20T02:53:24.708119+0000) 2023-07-20T02:53:33.717 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:33 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:33.219+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6820 osd.7 since back 2023-07-20T02:53:10.410163+0000 front 2023-07-20T02:53:10.410190+0000 (oldest deadline 2023-07-20T02:53:32.710073+0000) 2023-07-20T02:53:34.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:34 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:34.032+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:34.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:34 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:34.032+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:34.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:34 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:34.032+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:34.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:34 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:34.032+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:38.634686+0000 front 2023-07-20T02:52:38.634805+0000 (oldest deadline 2023-07-20T02:53:03.934673+0000) 2023-07-20T02:53:34.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:34 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:34.032+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:43.934952+0000 front 2023-07-20T02:52:43.934968+0000 (oldest deadline 2023-07-20T02:53:15.135593+0000) 2023-07-20T02:53:34.394 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:34 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:34.032+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:55.136272+0000 front 2023-07-20T02:52:55.136208+0000 (oldest deadline 2023-07-20T02:53:16.836201+0000) 2023-07-20T02:53:34.394 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:34 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:34.032+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2023-07-20T02:53:02.737385+0000 front 2023-07-20T02:53:02.737273+0000 (oldest deadline 2023-07-20T02:53:28.637249+0000) 2023-07-20T02:53:34.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:34 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:34.171+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:34.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:34 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:34.171+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:34.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:34 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:34.171+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:34.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:34 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:34.171+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.906632+0000 front 2023-07-20T02:52:48.906737+0000 (oldest deadline 2023-07-20T02:53:11.806525+0000) 2023-07-20T02:53:34.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:34 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:34.171+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:51.806912+0000 front 2023-07-20T02:52:51.806892+0000 (oldest deadline 2023-07-20T02:53:21.207783+0000) 2023-07-20T02:53:34.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:34 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:34.171+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2023-07-20T02:53:01.208430+0000 front 2023-07-20T02:53:01.208470+0000 (oldest deadline 2023-07-20T02:53:24.708119+0000) 2023-07-20T02:53:34.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:34 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:34.171+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6820 osd.7 since back 2023-07-20T02:53:10.410163+0000 front 2023-07-20T02:53:10.410190+0000 (oldest deadline 2023-07-20T02:53:32.710073+0000) 2023-07-20T02:53:35.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:35 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:35.021+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:14.429193+0000 front 2023-07-20T02:52:14.429229+0000 (oldest deadline 2023-07-20T02:52:37.929141+0000) 2023-07-20T02:53:35.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:35 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:35.021+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:25.331659+0000 front 2023-07-20T02:52:25.331725+0000 (oldest deadline 2023-07-20T02:52:47.031779+0000) 2023-07-20T02:53:35.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:35 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:35.021+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:30.433605+0000 front 2023-07-20T02:52:30.433525+0000 (oldest deadline 2023-07-20T02:52:58.634004+0000) 2023-07-20T02:53:35.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:35 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:35.021+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2023-07-20T02:52:38.634686+0000 front 2023-07-20T02:52:38.634805+0000 (oldest deadline 2023-07-20T02:53:03.934673+0000) 2023-07-20T02:53:35.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:35 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:35.021+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:43.934952+0000 front 2023-07-20T02:52:43.934968+0000 (oldest deadline 2023-07-20T02:53:15.135593+0000) 2023-07-20T02:53:35.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:35 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:35.021+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:55.136272+0000 front 2023-07-20T02:52:55.136208+0000 (oldest deadline 2023-07-20T02:53:16.836201+0000) 2023-07-20T02:53:35.393 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:35 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10[150454]: 2023-07-20T02:53:35.021+0000 7fddb3f9f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2023-07-20T02:53:02.737385+0000 front 2023-07-20T02:53:02.737273+0000 (oldest deadline 2023-07-20T02:53:28.637249+0000) 2023-07-20T02:53:35.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:35 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:35.173+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:35.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:35 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:35.173+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:35.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:35 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:35.173+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:35.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:35 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:35.173+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.906632+0000 front 2023-07-20T02:52:48.906737+0000 (oldest deadline 2023-07-20T02:53:11.806525+0000) 2023-07-20T02:53:35.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:35 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:35.173+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:51.806912+0000 front 2023-07-20T02:52:51.806892+0000 (oldest deadline 2023-07-20T02:53:21.207783+0000) 2023-07-20T02:53:35.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:35 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:35.173+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2023-07-20T02:53:01.208430+0000 front 2023-07-20T02:53:01.208470+0000 (oldest deadline 2023-07-20T02:53:24.708119+0000) 2023-07-20T02:53:35.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:35 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:35.173+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6820 osd.7 since back 2023-07-20T02:53:10.410163+0000 front 2023-07-20T02:53:10.410190+0000 (oldest deadline 2023-07-20T02:53:32.710073+0000) 2023-07-20T02:53:36.392 INFO:journalctl@ceph.osd.10.smithi158.stdout:Jul 20 02:53:35 smithi158 bash[156665]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-10 2023-07-20T02:53:36.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:36 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:36.160+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:36.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:36 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:36.160+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:36.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:36 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:36.160+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:36.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:36 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:36.160+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.906632+0000 front 2023-07-20T02:52:48.906737+0000 (oldest deadline 2023-07-20T02:53:11.806525+0000) 2023-07-20T02:53:36.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:36 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:36.160+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:51.806912+0000 front 2023-07-20T02:52:51.806892+0000 (oldest deadline 2023-07-20T02:53:21.207783+0000) 2023-07-20T02:53:36.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:36 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:36.160+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2023-07-20T02:53:01.208430+0000 front 2023-07-20T02:53:01.208470+0000 (oldest deadline 2023-07-20T02:53:24.708119+0000) 2023-07-20T02:53:36.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:36 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:36.160+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6820 osd.7 since back 2023-07-20T02:53:10.410163+0000 front 2023-07-20T02:53:10.410190+0000 (oldest deadline 2023-07-20T02:53:32.710073+0000) 2023-07-20T02:53:37.311 DEBUG:teuthology.orchestra.run.smithi158:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.10.service' 2023-07-20T02:53:37.349 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:53:37.350 INFO:tasks.cephadm.osd.10:Stopped osd.10 2023-07-20T02:53:37.350 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2023-07-20T02:53:37.351 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.11 2023-07-20T02:53:37.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:37 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:37.173+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:37.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:37 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:37.173+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:37.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:37 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:37.173+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:37.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:37 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:37.173+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.906632+0000 front 2023-07-20T02:52:48.906737+0000 (oldest deadline 2023-07-20T02:53:11.806525+0000) 2023-07-20T02:53:37.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:37 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:37.173+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:51.806912+0000 front 2023-07-20T02:52:51.806892+0000 (oldest deadline 2023-07-20T02:53:21.207783+0000) 2023-07-20T02:53:37.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:37 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:37.173+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2023-07-20T02:53:01.208430+0000 front 2023-07-20T02:53:01.208470+0000 (oldest deadline 2023-07-20T02:53:24.708119+0000) 2023-07-20T02:53:37.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:37 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:37.173+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6820 osd.7 since back 2023-07-20T02:53:10.410163+0000 front 2023-07-20T02:53:10.410190+0000 (oldest deadline 2023-07-20T02:53:32.710073+0000) 2023-07-20T02:53:37.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:37 smithi188 systemd[1]: Stopping Ceph osd.11 for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:53:37.965 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:37 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:37.698+0000 7f29f16ce700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:53:37.965 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:37 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:37.698+0000 7f29f16ce700 -1 osd.11 89 *** Got signal Terminated *** 2023-07-20T02:53:37.965 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:37 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:37.698+0000 7f29f16ce700 -1 osd.11 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T02:53:38.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:38 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:38.170+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:38.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:38 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:38.170+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:38.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:38 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:38.170+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:38.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:38 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:38.170+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.906632+0000 front 2023-07-20T02:52:48.906737+0000 (oldest deadline 2023-07-20T02:53:11.806525+0000) 2023-07-20T02:53:38.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:38 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:38.170+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:51.806912+0000 front 2023-07-20T02:52:51.806892+0000 (oldest deadline 2023-07-20T02:53:21.207783+0000) 2023-07-20T02:53:38.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:38 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:38.170+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2023-07-20T02:53:01.208430+0000 front 2023-07-20T02:53:01.208470+0000 (oldest deadline 2023-07-20T02:53:24.708119+0000) 2023-07-20T02:53:38.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:38 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:38.170+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6820 osd.7 since back 2023-07-20T02:53:10.410163+0000 front 2023-07-20T02:53:10.410190+0000 (oldest deadline 2023-07-20T02:53:32.710073+0000) 2023-07-20T02:53:39.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:39 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:39.174+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:39.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:39 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:39.174+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:39.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:39 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:39.174+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:39.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:39 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:39.174+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.906632+0000 front 2023-07-20T02:52:48.906737+0000 (oldest deadline 2023-07-20T02:53:11.806525+0000) 2023-07-20T02:53:39.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:39 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:39.174+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:51.806912+0000 front 2023-07-20T02:52:51.806892+0000 (oldest deadline 2023-07-20T02:53:21.207783+0000) 2023-07-20T02:53:39.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:39 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:39.174+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2023-07-20T02:53:01.208430+0000 front 2023-07-20T02:53:01.208470+0000 (oldest deadline 2023-07-20T02:53:24.708119+0000) 2023-07-20T02:53:39.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:39 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:39.174+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6820 osd.7 since back 2023-07-20T02:53:10.410163+0000 front 2023-07-20T02:53:10.410190+0000 (oldest deadline 2023-07-20T02:53:32.710073+0000) 2023-07-20T02:53:40.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:40 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:40.137+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:40.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:40 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:40.137+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:40.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:40 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:40.137+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:40.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:40 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:40.137+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.906632+0000 front 2023-07-20T02:52:48.906737+0000 (oldest deadline 2023-07-20T02:53:11.806525+0000) 2023-07-20T02:53:40.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:40 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:40.137+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:51.806912+0000 front 2023-07-20T02:52:51.806892+0000 (oldest deadline 2023-07-20T02:53:21.207783+0000) 2023-07-20T02:53:40.467 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:40 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:40.137+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2023-07-20T02:53:01.208430+0000 front 2023-07-20T02:53:01.208470+0000 (oldest deadline 2023-07-20T02:53:24.708119+0000) 2023-07-20T02:53:40.467 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:40 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:40.137+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6820 osd.7 since back 2023-07-20T02:53:10.410163+0000 front 2023-07-20T02:53:10.410190+0000 (oldest deadline 2023-07-20T02:53:32.710073+0000) 2023-07-20T02:53:41.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:41 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:41.097+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:41.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:41 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:41.097+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:41.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:41 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:41.097+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:41.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:41 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:41.097+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.906632+0000 front 2023-07-20T02:52:48.906737+0000 (oldest deadline 2023-07-20T02:53:11.806525+0000) 2023-07-20T02:53:41.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:41 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:41.097+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:51.806912+0000 front 2023-07-20T02:52:51.806892+0000 (oldest deadline 2023-07-20T02:53:21.207783+0000) 2023-07-20T02:53:41.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:41 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:41.097+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2023-07-20T02:53:01.208430+0000 front 2023-07-20T02:53:01.208470+0000 (oldest deadline 2023-07-20T02:53:24.708119+0000) 2023-07-20T02:53:41.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:41 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:41.097+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6820 osd.7 since back 2023-07-20T02:53:10.410163+0000 front 2023-07-20T02:53:10.410190+0000 (oldest deadline 2023-07-20T02:53:32.710073+0000) 2023-07-20T02:53:42.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:42 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:42.125+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-07-20T02:52:15.299408+0000 front 2023-07-20T02:52:15.299453+0000 (oldest deadline 2023-07-20T02:52:38.799446+0000) 2023-07-20T02:53:42.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:42 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:42.125+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6804 osd.1 since back 2023-07-20T02:52:21.601576+0000 front 2023-07-20T02:52:21.601445+0000 (oldest deadline 2023-07-20T02:52:47.501627+0000) 2023-07-20T02:53:42.465 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:42 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:42.125+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2023-07-20T02:52:28.602620+0000 front 2023-07-20T02:52:28.602648+0000 (oldest deadline 2023-07-20T02:52:55.003452+0000) 2023-07-20T02:53:42.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:42 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:42.125+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6812 osd.4 since back 2023-07-20T02:52:48.906632+0000 front 2023-07-20T02:52:48.906737+0000 (oldest deadline 2023-07-20T02:53:11.806525+0000) 2023-07-20T02:53:42.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:42 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:42.125+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2023-07-20T02:52:51.806912+0000 front 2023-07-20T02:52:51.806892+0000 (oldest deadline 2023-07-20T02:53:21.207783+0000) 2023-07-20T02:53:42.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:42 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:42.125+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2023-07-20T02:53:01.208430+0000 front 2023-07-20T02:53:01.208470+0000 (oldest deadline 2023-07-20T02:53:24.708119+0000) 2023-07-20T02:53:42.466 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:42 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11[151964]: 2023-07-20T02:53:42.125+0000 7f29ea4be700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.158:6820 osd.7 since back 2023-07-20T02:53:10.410163+0000 front 2023-07-20T02:53:10.410190+0000 (oldest deadline 2023-07-20T02:53:32.710073+0000) 2023-07-20T02:53:43.965 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 20 02:53:43 smithi188 bash[158405]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-osd-11 2023-07-20T02:53:44.948 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@osd.11.service' 2023-07-20T02:53:44.987 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:53:44.988 INFO:tasks.cephadm.osd.11:Stopped osd.11 2023-07-20T02:53:44.989 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2023-07-20T02:53:44.989 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.a 2023-07-20T02:53:45.450 INFO:journalctl@ceph.mds.a.smithi007.stdout:Jul 20 02:53:45 smithi007 systemd[1]: Stopping Ceph mds.a for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:53:45.950 INFO:journalctl@ceph.mds.a.smithi007.stdout:Jul 20 02:53:45 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-a[161388]: 2023-07-20T02:53:45.493+0000 7f6b87f13700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:53:45.950 INFO:journalctl@ceph.mds.a.smithi007.stdout:Jul 20 02:53:45 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-a[161388]: 2023-07-20T02:53:45.493+0000 7f6b87f13700 -1 mds.a *** got signal Terminated *** 2023-07-20T02:53:49.450 INFO:journalctl@ceph.mds.a.smithi007.stdout:Jul 20 02:53:49 smithi007 bash[167035]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-a 2023-07-20T02:53:49.548 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.a.service' 2023-07-20T02:53:49.636 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:53:49.637 INFO:tasks.cephadm.mds.a:Stopped mds.a 2023-07-20T02:53:49.637 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2023-07-20T02:53:49.638 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.d 2023-07-20T02:53:49.950 INFO:journalctl@ceph.mds.d.smithi007.stdout:Jul 20 02:53:49 smithi007 systemd[1]: Stopping Ceph mds.d for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:53:50.450 INFO:journalctl@ceph.mds.d.smithi007.stdout:Jul 20 02:53:50 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-d[160385]: 2023-07-20T02:53:50.125+0000 7f162b53e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:53:50.450 INFO:journalctl@ceph.mds.d.smithi007.stdout:Jul 20 02:53:50 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-d[160385]: 2023-07-20T02:53:50.125+0000 7f162b53e700 -1 mds.d *** got signal Terminated *** 2023-07-20T02:53:54.700 INFO:journalctl@ceph.mds.d.smithi007.stdout:Jul 20 02:53:54 smithi007 bash[167181]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-d 2023-07-20T02:53:54.821 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.d.service' 2023-07-20T02:53:54.906 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:53:54.906 INFO:tasks.cephadm.mds.d:Stopped mds.d 2023-07-20T02:53:54.907 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2023-07-20T02:53:54.907 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.g 2023-07-20T02:53:55.200 INFO:journalctl@ceph.mds.g.smithi007.stdout:Jul 20 02:53:54 smithi007 systemd[1]: Stopping Ceph mds.g for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:53:55.700 INFO:journalctl@ceph.mds.g.smithi007.stdout:Jul 20 02:53:55 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-g[160997]: 2023-07-20T02:53:55.390+0000 7ff04e26c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:53:55.700 INFO:journalctl@ceph.mds.g.smithi007.stdout:Jul 20 02:53:55 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-g[160997]: 2023-07-20T02:53:55.390+0000 7ff04e26c700 -1 mds.g *** got signal Terminated *** 2023-07-20T02:53:57.700 INFO:journalctl@ceph.mds.g.smithi007.stdout:Jul 20 02:53:57 smithi007 bash[167323]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-g 2023-07-20T02:53:57.974 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.g.service' 2023-07-20T02:53:58.058 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:53:58.059 INFO:tasks.cephadm.mds.g:Stopped mds.g 2023-07-20T02:53:58.059 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2023-07-20T02:53:58.060 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.j 2023-07-20T02:53:58.450 INFO:journalctl@ceph.mds.j.smithi007.stdout:Jul 20 02:53:58 smithi007 systemd[1]: Stopping Ceph mds.j for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:53:58.950 INFO:journalctl@ceph.mds.j.smithi007.stdout:Jul 20 02:53:58 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-j[159994]: 2023-07-20T02:53:58.549+0000 7f2cf4aac700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:53:58.950 INFO:journalctl@ceph.mds.j.smithi007.stdout:Jul 20 02:53:58 smithi007 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-j[159994]: 2023-07-20T02:53:58.549+0000 7f2cf4aac700 -1 mds.j *** got signal Terminated *** 2023-07-20T02:54:03.753 INFO:journalctl@ceph.mds.j.smithi007.stdout:Jul 20 02:54:03 smithi007 bash[167470]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-j 2023-07-20T02:54:04.056 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.j.service' 2023-07-20T02:54:04.141 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:54:04.143 INFO:tasks.cephadm.mds.j:Stopped mds.j 2023-07-20T02:54:04.143 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2023-07-20T02:54:04.144 DEBUG:teuthology.orchestra.run.smithi158:> sudo systemctl stop ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.b 2023-07-20T02:54:04.642 INFO:journalctl@ceph.mds.b.smithi158.stdout:Jul 20 02:54:04 smithi158 systemd[1]: Stopping Ceph mds.b for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:54:05.144 INFO:journalctl@ceph.mds.b.smithi158.stdout:Jul 20 02:54:04 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-b[154225]: 2023-07-20T02:54:04.667+0000 7f091f127700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:54:05.144 INFO:journalctl@ceph.mds.b.smithi158.stdout:Jul 20 02:54:04 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-b[154225]: 2023-07-20T02:54:04.667+0000 7f091f127700 -1 mds.b *** got signal Terminated *** 2023-07-20T02:54:14.963 INFO:journalctl@ceph.mds.b.smithi158.stdout:Jul 20 02:54:14 smithi158 bash[156922]: time="2023-07-20T02:54:14Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-b in 10 seconds, resorting to SIGKILL" 2023-07-20T02:54:14.963 INFO:journalctl@ceph.mds.b.smithi158.stdout:Jul 20 02:54:14 smithi158 bash[156922]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-b 2023-07-20T02:54:15.206 DEBUG:teuthology.orchestra.run.smithi158:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.b.service' 2023-07-20T02:54:15.234 INFO:journalctl@ceph.mds.b.smithi158.stdout:Jul 20 02:54:15 smithi158 systemd[1]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.b.service: Main process exited, code=exited, status=137/n/a 2023-07-20T02:54:15.235 INFO:journalctl@ceph.mds.b.smithi158.stdout:Jul 20 02:54:15 smithi158 systemd[1]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.b.service: Failed with result 'exit-code'. 2023-07-20T02:54:15.235 INFO:journalctl@ceph.mds.b.smithi158.stdout:Jul 20 02:54:15 smithi158 systemd[1]: Stopped Ceph mds.b for 1b04643c-26a7-11ee-9b34-001a4aab830c. 2023-07-20T02:54:15.247 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:54:15.248 INFO:tasks.cephadm.mds.b:Stopped mds.b 2023-07-20T02:54:15.249 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2023-07-20T02:54:15.249 DEBUG:teuthology.orchestra.run.smithi158:> sudo systemctl stop ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.e 2023-07-20T02:54:15.611 INFO:journalctl@ceph.mds.e.smithi158.stdout:Jul 20 02:54:15 smithi158 systemd[1]: Stopping Ceph mds.e for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:54:15.892 INFO:journalctl@ceph.mds.e.smithi158.stdout:Jul 20 02:54:15 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-e[153533]: 2023-07-20T02:54:15.595+0000 7f1e49bdc700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:54:15.893 INFO:journalctl@ceph.mds.e.smithi158.stdout:Jul 20 02:54:15 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-e[153533]: 2023-07-20T02:54:15.595+0000 7f1e49bdc700 -1 mds.e *** got signal Terminated *** 2023-07-20T02:54:18.032 INFO:journalctl@ceph.mds.e.smithi158.stdout:Jul 20 02:54:17 smithi158 bash[157189]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-e 2023-07-20T02:54:18.039 DEBUG:teuthology.orchestra.run.smithi158:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.e.service' 2023-07-20T02:54:18.078 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:54:18.079 INFO:tasks.cephadm.mds.e:Stopped mds.e 2023-07-20T02:54:18.080 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2023-07-20T02:54:18.081 DEBUG:teuthology.orchestra.run.smithi158:> sudo systemctl stop ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.h 2023-07-20T02:54:18.375 INFO:journalctl@ceph.mds.h.smithi158.stdout:Jul 20 02:54:18 smithi158 systemd[1]: Stopping Ceph mds.h for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:54:18.642 INFO:journalctl@ceph.mds.h.smithi158.stdout:Jul 20 02:54:18 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-h[153875]: 2023-07-20T02:54:18.445+0000 7f40f84bd700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:54:18.643 INFO:journalctl@ceph.mds.h.smithi158.stdout:Jul 20 02:54:18 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-h[153875]: 2023-07-20T02:54:18.445+0000 7f40f84bd700 -1 mds.h *** got signal Terminated *** 2023-07-20T02:54:20.637 INFO:journalctl@ceph.mds.h.smithi158.stdout:Jul 20 02:54:20 smithi158 bash[157339]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-h 2023-07-20T02:54:20.643 DEBUG:teuthology.orchestra.run.smithi158:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.h.service' 2023-07-20T02:54:20.681 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:54:20.681 INFO:tasks.cephadm.mds.h:Stopped mds.h 2023-07-20T02:54:20.682 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2023-07-20T02:54:20.682 DEBUG:teuthology.orchestra.run.smithi158:> sudo systemctl stop ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.k 2023-07-20T02:54:20.893 INFO:journalctl@ceph.mds.k.smithi158.stdout:Jul 20 02:54:20 smithi158 systemd[1]: Stopping Ceph mds.k for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:54:21.392 INFO:journalctl@ceph.mds.k.smithi158.stdout:Jul 20 02:54:21 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-k[153190]: 2023-07-20T02:54:21.063+0000 7f3397a38700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:54:21.396 INFO:journalctl@ceph.mds.k.smithi158.stdout:Jul 20 02:54:21 smithi158 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-k[153190]: 2023-07-20T02:54:21.063+0000 7f3397a38700 -1 mds.k *** got signal Terminated *** 2023-07-20T02:54:24.893 INFO:journalctl@ceph.mds.k.smithi158.stdout:Jul 20 02:54:24 smithi158 bash[157498]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-k 2023-07-20T02:54:24.966 DEBUG:teuthology.orchestra.run.smithi158:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.k.service' 2023-07-20T02:54:25.003 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:54:25.003 INFO:tasks.cephadm.mds.k:Stopped mds.k 2023-07-20T02:54:25.004 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2023-07-20T02:54:25.004 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.c 2023-07-20T02:54:25.410 INFO:journalctl@ceph.mds.c.smithi188.stdout:Jul 20 02:54:25 smithi188 systemd[1]: Stopping Ceph mds.c for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:54:25.716 INFO:journalctl@ceph.mds.c.smithi188.stdout:Jul 20 02:54:25 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-c[155640]: 2023-07-20T02:54:25.394+0000 7f7827f4f700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:54:25.716 INFO:journalctl@ceph.mds.c.smithi188.stdout:Jul 20 02:54:25 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-c[155640]: 2023-07-20T02:54:25.394+0000 7f7827f4f700 -1 mds.c *** got signal Terminated *** 2023-07-20T02:54:30.647 INFO:journalctl@ceph.mds.c.smithi188.stdout:Jul 20 02:54:30 smithi188 bash[158676]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-c 2023-07-20T02:54:30.833 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.c.service' 2023-07-20T02:54:30.871 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:54:30.872 INFO:tasks.cephadm.mds.c:Stopped mds.c 2023-07-20T02:54:30.873 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2023-07-20T02:54:30.873 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.f 2023-07-20T02:54:31.215 INFO:journalctl@ceph.mds.f.smithi188.stdout:Jul 20 02:54:30 smithi188 systemd[1]: Stopping Ceph mds.f for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:54:31.716 INFO:journalctl@ceph.mds.f.smithi188.stdout:Jul 20 02:54:31 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-f[155302]: 2023-07-20T02:54:31.252+0000 7fb152c0b700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:54:31.716 INFO:journalctl@ceph.mds.f.smithi188.stdout:Jul 20 02:54:31 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-f[155302]: 2023-07-20T02:54:31.252+0000 7fb152c0b700 -1 mds.f *** got signal Terminated *** 2023-07-20T02:54:36.010 INFO:journalctl@ceph.mds.f.smithi188.stdout:Jul 20 02:54:35 smithi188 bash[158811]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-f 2023-07-20T02:54:36.015 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.f.service' 2023-07-20T02:54:36.051 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:54:36.052 INFO:tasks.cephadm.mds.f:Stopped mds.f 2023-07-20T02:54:36.052 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2023-07-20T02:54:36.053 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.i 2023-07-20T02:54:36.466 INFO:journalctl@ceph.mds.i.smithi188.stdout:Jul 20 02:54:36 smithi188 systemd[1]: Stopping Ceph mds.i for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:54:36.966 INFO:journalctl@ceph.mds.i.smithi188.stdout:Jul 20 02:54:36 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-i[155981]: 2023-07-20T02:54:36.459+0000 7fe7c22d4700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:54:36.966 INFO:journalctl@ceph.mds.i.smithi188.stdout:Jul 20 02:54:36 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-i[155981]: 2023-07-20T02:54:36.459+0000 7fe7c22d4700 -1 mds.i *** got signal Terminated *** 2023-07-20T02:54:41.963 INFO:journalctl@ceph.mds.i.smithi188.stdout:Jul 20 02:54:41 smithi188 bash[158951]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-i 2023-07-20T02:54:41.968 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.i.service' 2023-07-20T02:54:42.002 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:54:42.003 INFO:tasks.cephadm.mds.i:Stopped mds.i 2023-07-20T02:54:42.003 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2023-07-20T02:54:42.003 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.l 2023-07-20T02:54:42.381 INFO:journalctl@ceph.mds.l.smithi188.stdout:Jul 20 02:54:42 smithi188 systemd[1]: Stopping Ceph mds.l for 1b04643c-26a7-11ee-9b34-001a4aab830c... 2023-07-20T02:54:42.715 INFO:journalctl@ceph.mds.l.smithi188.stdout:Jul 20 02:54:42 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-l[154957]: 2023-07-20T02:54:42.364+0000 7efc064f1700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T02:54:42.716 INFO:journalctl@ceph.mds.l.smithi188.stdout:Jul 20 02:54:42 smithi188 ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-l[154957]: 2023-07-20T02:54:42.364+0000 7efc064f1700 -1 mds.l *** got signal Terminated *** 2023-07-20T02:54:46.955 INFO:journalctl@ceph.mds.l.smithi188.stdout:Jul 20 02:54:46 smithi188 bash[159093]: ceph-1b04643c-26a7-11ee-9b34-001a4aab830c-mds-l 2023-07-20T02:54:47.162 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b04643c-26a7-11ee-9b34-001a4aab830c@mds.l.service' 2023-07-20T02:54:47.197 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T02:54:47.198 INFO:tasks.cephadm.mds.l:Stopped mds.l 2023-07-20T02:54:47.199 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c --force --keep-logs 2023-07-20T02:54:47.599 INFO:teuthology.orchestra.run.smithi007.stdout:Deleting cluster with fsid: 1b04643c-26a7-11ee-9b34-001a4aab830c 2023-07-20T02:54:50.970 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c --force --keep-logs 2023-07-20T02:54:51.276 INFO:teuthology.orchestra.run.smithi158.stdout:Deleting cluster with fsid: 1b04643c-26a7-11ee-9b34-001a4aab830c 2023-07-20T02:54:52.838 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c --force --keep-logs 2023-07-20T02:54:53.137 INFO:teuthology.orchestra.run.smithi188.stdout:Deleting cluster with fsid: 1b04643c-26a7-11ee-9b34-001a4aab830c 2023-07-20T02:54:54.684 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T02:54:54.764 DEBUG:teuthology.orchestra.run.smithi158:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T02:54:54.801 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T02:54:54.830 INFO:tasks.cephadm:Archiving crash dumps... 2023-07-20T02:54:54.832 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344290/remote/smithi007/crash 2023-07-20T02:54:54.833 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/crash -- . 2023-07-20T02:54:54.919 INFO:teuthology.orchestra.run.smithi007.stderr:tar: /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-20T02:54:54.919 INFO:teuthology.orchestra.run.smithi007.stderr:tar: Error is not recoverable: exiting now 2023-07-20T02:54:54.922 DEBUG:teuthology.misc:Transferring archived files from smithi158:/var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344290/remote/smithi158/crash 2023-07-20T02:54:54.923 DEBUG:teuthology.orchestra.run.smithi158:> sudo tar c -f - -C /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/crash -- . 2023-07-20T02:54:55.032 INFO:teuthology.orchestra.run.smithi158.stderr:tar: /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-20T02:54:55.032 INFO:teuthology.orchestra.run.smithi158.stderr:tar: Error is not recoverable: exiting now 2023-07-20T02:54:55.033 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344290/remote/smithi188/crash 2023-07-20T02:54:55.035 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/crash -- . 2023-07-20T02:54:55.076 INFO:teuthology.orchestra.run.smithi188.stderr:tar: /var/lib/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-20T02:54:55.076 INFO:teuthology.orchestra.run.smithi188.stderr:tar: Error is not recoverable: exiting now 2023-07-20T02:54:55.077 INFO:tasks.cephadm:Checking cluster log for badness... 2023-07-20T02:54:55.078 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-07-20T02:54:55.156 INFO:teuthology.orchestra.run.smithi007.stderr:grep: /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph.log: No such file or directory 2023-07-20T02:54:55.161 INFO:tasks.cephadm:Compressing logs... 2023-07-20T02:54:55.161 DEBUG:teuthology.orchestra.run.smithi007:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-20T02:54:55.167 DEBUG:teuthology.orchestra.run.smithi158:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-20T02:54:55.170 DEBUG:teuthology.orchestra.run.smithi188:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-20T02:54:55.197 INFO:teuthology.orchestra.run.smithi188.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-20T02:54:55.197 INFO:teuthology.orchestra.run.smithi188.stderr:: No such file or directory 2023-07-20T02:54:55.197 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.5.log 2023-07-20T02:54:55.198 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.2.log 2023-07-20T02:54:55.199 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.11.log 2023-07-20T02:54:55.199 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mgr.z.log 2023-07-20T02:54:55.199 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.8.log 2023-07-20T02:54:55.199 INFO:teuthology.orchestra.run.smithi188.stderr: 92.0% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-20T02:54:55.200 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.f.log 2023-07-20T02:54:55.200 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.c.log 2023-07-20T02:54:55.201 INFO:teuthology.orchestra.run.smithi158.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-20T02:54:55.201 INFO:teuthology.orchestra.run.smithi158.stderr:: No such file or directory 2023-07-20T02:54:55.201 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.10.log 2023-07-20T02:54:55.201 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.h.log 2023-07-20T02:54:55.201 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.e.log 2023-07-20T02:54:55.202 INFO:teuthology.orchestra.run.smithi158.stderr:gzip /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.h.log: -5 --verbose -- /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.4.log 2023-07-20T02:54:55.202 INFO:teuthology.orchestra.run.smithi158.stderr: 91.6% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-20T02:54:55.202 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.1.log 2023-07-20T02:54:55.203 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.k.log 2023-07-20T02:54:55.203 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.f.log: /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.c.log: 91.4% -- replaced with /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mgr.z.log.gz 2023-07-20T02:54:55.204 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.i.log 2023-07-20T02:54:55.204 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mon.c.log 2023-07-20T02:54:55.206 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.1.log: /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.k.log: 93.5% -- replaced with /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.h.log.gz 2023-07-20T02:54:55.207 INFO:teuthology.orchestra.run.smithi158.stderr: 94.1% -- replaced with /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.e.log.gz 2023-07-20T02:54:55.207 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-volume.log 2023-07-20T02:54:55.207 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.7.log 2023-07-20T02:54:55.211 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-volume.log: /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.7.log: 93.5% -- replaced with /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.k.log.gz 2023-07-20T02:54:55.211 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mon.b.log 2023-07-20T02:54:55.212 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.b.log 2023-07-20T02:54:55.213 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mon.c.log: 94.2% -- replaced with /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.f.log.gz 2023-07-20T02:54:55.213 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-volume.log 2023-07-20T02:54:55.213 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-volume.log: 93.6% -- replaced with /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.c.log.gz 2023-07-20T02:54:55.213 INFO:teuthology.orchestra.run.smithi188.stderr: 93.6% -- replaced with /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.i.log.gz 2023-07-20T02:54:55.213 INFO:teuthology.orchestra.run.smithi188.stderr: 91.5% -- replaced with /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-volume.log.gz 2023-07-20T02:54:55.213 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.l.log 2023-07-20T02:54:55.215 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mon.b.log: /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.b.log: 91.4% -- replaced with /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-volume.log.gz 2023-07-20T02:54:55.215 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mgr.y.log 2023-07-20T02:54:55.217 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.l.log: 94.2% -- replaced with /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.l.log.gz 2023-07-20T02:54:55.220 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mgr.y.log: 91.4% -- replaced with /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mgr.y.log.gz 2023-07-20T02:54:55.231 INFO:teuthology.orchestra.run.smithi007.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-07-20T02:54:55.231 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-20T02:54:55.232 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2023-07-20T02:54:55.233 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.0.log 2023-07-20T02:54:55.234 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.log: gzip -5 --verbose -- /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.g.log 2023-07-20T02:54:55.235 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.a.log 2023-07-20T02:54:55.236 INFO:teuthology.orchestra.run.smithi007.stderr: 93.9% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2023-07-20T02:54:55.237 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.6.log 2023-07-20T02:54:55.237 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.a.log: 94.9% -- replaced with /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.g.log.gz 2023-07-20T02:54:55.238 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.3.log 2023-07-20T02:54:55.239 INFO:teuthology.orchestra.run.smithi007.stderr: 92.9% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-20T02:54:55.240 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.d.log 2023-07-20T02:54:55.241 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mgr.x.log 2023-07-20T02:54:55.241 INFO:teuthology.orchestra.run.smithi007.stderr: 94.0% -- replaced with /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.a.log.gz 2023-07-20T02:54:55.242 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.9.log 2023-07-20T02:54:55.243 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mgr.x.log: 94.8% -- replaced with /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.d.log.gz 2023-07-20T02:54:55.244 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-volume.log 2023-07-20T02:54:55.245 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.j.log 2023-07-20T02:54:55.247 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mon.a.log 2023-07-20T02:54:55.250 INFO:teuthology.orchestra.run.smithi158.stderr: 93.7% -- replaced with /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.b.log.gz 2023-07-20T02:54:55.256 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.j.log: /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mon.a.log: 91.5% -- replaced with /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-volume.log.gz 2023-07-20T02:54:55.256 INFO:teuthology.orchestra.run.smithi007.stderr: 93.3% -- replaced with /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mds.j.log.gz 2023-07-20T02:54:55.345 INFO:teuthology.orchestra.run.smithi188.stderr: 92.7% -- replaced with /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.11.log.gz 2023-07-20T02:54:55.374 INFO:teuthology.orchestra.run.smithi158.stderr: 92.3% -- replaced with /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mon.b.log.gz 2023-07-20T02:54:55.380 INFO:teuthology.orchestra.run.smithi158.stderr: 92.2% -- replaced with /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.4.log.gz 2023-07-20T02:54:55.386 INFO:teuthology.orchestra.run.smithi007.stderr: 88.8% -- replaced with /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mgr.x.log.gz 2023-07-20T02:54:55.386 INFO:teuthology.orchestra.run.smithi158.stderr: 92.2% -- replaced with /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.10.log.gz 2023-07-20T02:54:55.400 INFO:teuthology.orchestra.run.smithi158.stderr: 92.1% -- replaced with /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.7.log.gz 2023-07-20T02:54:55.400 INFO:teuthology.orchestra.run.smithi188.stderr: 91.9% -- replaced with /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mon.c.log.gz 2023-07-20T02:54:55.409 INFO:teuthology.orchestra.run.smithi188.stderr: 92.0% -- replaced with /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.5.log.gz 2023-07-20T02:54:55.428 INFO:teuthology.orchestra.run.smithi188.stderr: 92.2% -- replaced with /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.8.log.gz 2023-07-20T02:54:55.440 INFO:teuthology.orchestra.run.smithi007.stderr: 92.0% -- replaced with /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.3.log.gz 2023-07-20T02:54:55.452 INFO:teuthology.orchestra.run.smithi188.stderr: 92.0% -- replaced with /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.2.log.gz 2023-07-20T02:54:55.454 INFO:teuthology.orchestra.run.smithi188.stderr: 2023-07-20T02:54:55.454 INFO:teuthology.orchestra.run.smithi188.stderr:real 0m0.267s 2023-07-20T02:54:55.454 INFO:teuthology.orchestra.run.smithi188.stderr:user 0m1.045s 2023-07-20T02:54:55.454 INFO:teuthology.orchestra.run.smithi188.stderr:sys 0m0.057s 2023-07-20T02:54:55.461 INFO:teuthology.orchestra.run.smithi007.stderr: 92.2% -- replaced with /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.9.log.gz 2023-07-20T02:54:55.496 INFO:teuthology.orchestra.run.smithi007.stderr: 92.1% -- replaced with /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.6.log.gz 2023-07-20T02:54:55.500 INFO:teuthology.orchestra.run.smithi158.stderr: 91.9% -- replaced with /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.1.log.gz 2023-07-20T02:54:55.502 INFO:teuthology.orchestra.run.smithi158.stderr: 2023-07-20T02:54:55.502 INFO:teuthology.orchestra.run.smithi158.stderr:real 0m0.315s 2023-07-20T02:54:55.502 INFO:teuthology.orchestra.run.smithi158.stderr:user 0m1.055s 2023-07-20T02:54:55.502 INFO:teuthology.orchestra.run.smithi158.stderr:sys 0m0.060s 2023-07-20T02:54:55.524 INFO:teuthology.orchestra.run.smithi007.stderr: 92.1% -- replaced with /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-osd.0.log.gz 2023-07-20T02:54:55.604 INFO:teuthology.orchestra.run.smithi007.stderr: 91.0% -- replaced with /var/log/ceph/1b04643c-26a7-11ee-9b34-001a4aab830c/ceph-mon.a.log.gz 2023-07-20T02:54:55.607 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-07-20T02:54:55.607 INFO:teuthology.orchestra.run.smithi007.stderr:real 0m0.404s 2023-07-20T02:54:55.607 INFO:teuthology.orchestra.run.smithi007.stderr:user 0m1.378s 2023-07-20T02:54:55.608 INFO:teuthology.orchestra.run.smithi007.stderr:sys 0m0.200s 2023-07-20T02:54:55.608 INFO:tasks.cephadm:Archiving logs... 2023-07-20T02:54:55.609 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344290/remote/smithi007/log 2023-07-20T02:54:55.610 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-20T02:54:55.864 DEBUG:teuthology.misc:Transferring archived files from smithi158:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344290/remote/smithi158/log 2023-07-20T02:54:55.865 DEBUG:teuthology.orchestra.run.smithi158:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-20T02:54:56.026 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344290/remote/smithi188/log 2023-07-20T02:54:56.027 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-20T02:54:56.192 INFO:tasks.cephadm:Removing cluster... 2023-07-20T02:54:56.193 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c --force 2023-07-20T02:54:56.579 INFO:teuthology.orchestra.run.smithi007.stdout:Deleting cluster with fsid: 1b04643c-26a7-11ee-9b34-001a4aab830c 2023-07-20T02:54:57.302 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c --force 2023-07-20T02:54:57.609 INFO:teuthology.orchestra.run.smithi158.stdout:Deleting cluster with fsid: 1b04643c-26a7-11ee-9b34-001a4aab830c 2023-07-20T02:54:58.061 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1b04643c-26a7-11ee-9b34-001a4aab830c --force 2023-07-20T02:54:58.361 INFO:teuthology.orchestra.run.smithi188.stdout:Deleting cluster with fsid: 1b04643c-26a7-11ee-9b34-001a4aab830c 2023-07-20T02:54:58.809 INFO:tasks.cephadm:Removing cephadm ... 2023-07-20T02:54:58.809 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-20T02:54:58.860 DEBUG:teuthology.orchestra.run.smithi158:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-20T02:54:58.881 DEBUG:teuthology.orchestra.run.smithi188:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-20T02:54:58.899 INFO:tasks.cephadm:Teardown complete 2023-07-20T02:54:58.900 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephadm.py", line 1643, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephadm.py", line 1558, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephadm.py", line 980, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/filesystem.py", line 535, in __init__ self.create() File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/filesystem.py", line 773, in create subvol_path = self.get_ceph_cmd_stdout( File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/cephfs_test_case.py", line 73, in get_ceph_cmd_stdout return self.run_ceph_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/cephfs_test_case.py", line 58, in run_ceph_cmd return self.mon_manager.run_cluster_cmd(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 2: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name=qa sv_0' 2023-07-20T02:54:58.986 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=6390ba8843e247af9294e3a42eb98e38 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephadm.py", line 1643, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephadm.py", line 1558, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephadm.py", line 980, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/filesystem.py", line 535, in __init__ self.create() File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/filesystem.py", line 773, in create subvol_path = self.get_ceph_cmd_stdout( File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/cephfs_test_case.py", line 73, in get_ceph_cmd_stdout return self.run_ceph_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/cephfs_test_case.py", line 58, in run_ceph_cmd return self.mon_manager.run_cluster_cmd(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 2: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name=qa sv_0' 2023-07-20T02:54:58.991 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-07-20T02:54:59.003 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-07-20T02:54:59.020 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-07-20T02:54:59.021 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-20T02:54:59.111 DEBUG:teuthology.orchestra.run.smithi158:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-20T02:54:59.115 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-20T02:54:59.261 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-07-20T02:54:59.271 INFO:teuthology.task.clock:Checking final clock skew... 2023-07-20T02:54:59.271 DEBUG:teuthology.orchestra.run.smithi007:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T02:54:59.276 DEBUG:teuthology.orchestra.run.smithi158:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T02:54:59.279 DEBUG:teuthology.orchestra.run.smithi188:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T02:54:59.294 INFO:teuthology.orchestra.run.smithi188.stderr:bash: ntpq: command not found 2023-07-20T02:54:59.295 INFO:teuthology.orchestra.run.smithi158.stderr:bash: ntpq: command not found 2023-07-20T02:54:59.299 INFO:teuthology.orchestra.run.smithi188.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T02:54:59.299 INFO:teuthology.orchestra.run.smithi188.stdout:=============================================================================== 2023-07-20T02:54:59.299 INFO:teuthology.orchestra.run.smithi188.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 17 -1727us[-1699us] +/- 86ms 2023-07-20T02:54:59.300 INFO:teuthology.orchestra.run.smithi188.stdout:^* hv02.front.sepia.ceph.com 3 6 377 16 +2429us[+2457us] +/- 75ms 2023-07-20T02:54:59.300 INFO:teuthology.orchestra.run.smithi188.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 14 -1151us[-1151us] +/- 88ms 2023-07-20T02:54:59.300 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T02:54:59.301 INFO:teuthology.orchestra.run.smithi158.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T02:54:59.302 INFO:teuthology.orchestra.run.smithi158.stdout:=============================================================================== 2023-07-20T02:54:59.302 INFO:teuthology.orchestra.run.smithi158.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 18 -1719us[-1726us] +/- 86ms 2023-07-20T02:54:59.302 INFO:teuthology.orchestra.run.smithi158.stdout:^* hv02.front.sepia.ceph.com 3 7 377 15 +2433us[+2427us] +/- 75ms 2023-07-20T02:54:59.302 INFO:teuthology.orchestra.run.smithi158.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 15 -1154us[-1154us] +/- 88ms 2023-07-20T02:54:59.302 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T02:54:59.315 INFO:teuthology.orchestra.run.smithi007.stderr:bash: ntpq: command not found 2023-07-20T02:54:59.323 INFO:teuthology.orchestra.run.smithi007.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T02:54:59.324 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================== 2023-07-20T02:54:59.324 INFO:teuthology.orchestra.run.smithi007.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 15 -1709us[-1713us] +/- 86ms 2023-07-20T02:54:59.324 INFO:teuthology.orchestra.run.smithi007.stdout:^* hv02.front.sepia.ceph.com 3 7 377 13 +2436us[+2432us] +/- 75ms 2023-07-20T02:54:59.325 INFO:teuthology.orchestra.run.smithi007.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 15 -1136us[-1140us] +/- 88ms 2023-07-20T02:54:59.325 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T02:54:59.326 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-07-20T02:54:59.345 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-07-20T02:54:59.346 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-07-20T02:54:59.355 DEBUG:teuthology.orchestra.run.smithi007:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-20T02:54:59.541 DEBUG:teuthology.orchestra.run.smithi158:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-20T02:54:59.641 DEBUG:teuthology.orchestra.run.smithi188:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-20T02:54:59.733 DEBUG:teuthology.orchestra.run.smithi007:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T02:54:59.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T02:54:59.811 DEBUG:teuthology.orchestra.run.smithi158:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T02:54:59.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T02:54:59.847 DEBUG:teuthology.orchestra.run.smithi188:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T02:54:59.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T02:54:59.883 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-07-20T02:54:59.894 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-07-20T02:54:59.912 INFO:teuthology.task.internal:Duration was 1899.205129 seconds 2023-07-20T02:54:59.912 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-07-20T02:54:59.922 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-07-20T02:54:59.923 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-20T02:54:59.928 DEBUG:teuthology.orchestra.run.smithi158:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-20T02:54:59.931 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-20T02:54:59.976 INFO:teuthology.orchestra.run.smithi158.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T02:54:59.977 INFO:teuthology.orchestra.run.smithi188.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T02:55:00.043 INFO:teuthology.orchestra.run.smithi007.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T02:55:00.228 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-07-20T02:55:00.229 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi007.front.sepia.ceph.com 2023-07-20T02:55:00.229 DEBUG:teuthology.orchestra.run.smithi007:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-20T02:55:00.288 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi158.front.sepia.ceph.com 2023-07-20T02:55:00.289 DEBUG:teuthology.orchestra.run.smithi158:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-20T02:55:00.311 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi188.front.sepia.ceph.com 2023-07-20T02:55:00.312 DEBUG:teuthology.orchestra.run.smithi188:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-20T02:55:00.334 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-07-20T02:55:00.334 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-20T02:55:00.339 DEBUG:teuthology.orchestra.run.smithi158:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-20T02:55:00.353 DEBUG:teuthology.orchestra.run.smithi188:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-20T02:55:00.430 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-07-20T02:55:00.430 DEBUG:teuthology.orchestra.run.smithi007:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-20T02:55:00.434 DEBUG:teuthology.orchestra.run.smithi158:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-20T02:55:00.436 DEBUG:teuthology.orchestra.run.smithi188:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-20T02:55:00.582 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-07-20T02:55:00.595 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-07-20T02:55:00.596 DEBUG:teuthology.orchestra.run.smithi007:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-20T02:55:00.665 DEBUG:teuthology.orchestra.run.smithi158:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-20T02:55:00.693 DEBUG:teuthology.orchestra.run.smithi188:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-20T02:55:00.721 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-07-20T02:55:00.731 DEBUG:teuthology.orchestra.run.smithi007:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-20T02:55:00.736 DEBUG:teuthology.orchestra.run.smithi158:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-20T02:55:00.738 DEBUG:teuthology.orchestra.run.smithi188:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-20T02:55:00.761 INFO:teuthology.orchestra.run.smithi158.stdout:kernel.core_pattern = core 2023-07-20T02:55:00.787 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = core 2023-07-20T02:55:00.800 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = core 2023-07-20T02:55:00.838 DEBUG:teuthology.orchestra.run.smithi007:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-20T02:55:00.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T02:55:00.881 DEBUG:teuthology.orchestra.run.smithi158:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-20T02:55:00.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T02:55:00.899 DEBUG:teuthology.orchestra.run.smithi188:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-20T02:55:00.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T02:55:00.917 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-07-20T02:55:00.933 INFO:teuthology.task.internal:Transferring archived files... 2023-07-20T02:55:00.934 DEBUG:teuthology.misc:Transferring archived files from smithi007:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344290/remote/smithi007 2023-07-20T02:55:00.935 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-20T02:55:01.015 DEBUG:teuthology.misc:Transferring archived files from smithi158:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344290/remote/smithi158 2023-07-20T02:55:01.016 DEBUG:teuthology.orchestra.run.smithi158:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-20T02:55:01.054 DEBUG:teuthology.misc:Transferring archived files from smithi188:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344290/remote/smithi188 2023-07-20T02:55:01.055 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-20T02:55:01.091 INFO:teuthology.task.internal:Removing archive directory... 2023-07-20T02:55:01.092 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-20T02:55:01.097 DEBUG:teuthology.orchestra.run.smithi158:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-20T02:55:01.099 DEBUG:teuthology.orchestra.run.smithi188:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-20T02:55:01.142 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-07-20T02:55:01.160 INFO:teuthology.task.internal:Not uploading archives. 2023-07-20T02:55:01.160 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-07-20T02:55:01.169 INFO:teuthology.task.internal:Tidying up after the test... 2023-07-20T02:55:01.170 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-20T02:55:01.181 DEBUG:teuthology.orchestra.run.smithi158:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-20T02:55:01.183 DEBUG:teuthology.orchestra.run.smithi188:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-20T02:55:01.197 INFO:teuthology.orchestra.run.smithi158.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 20 02:55 /home/ubuntu/cephtest 2023-07-20T02:55:01.200 INFO:teuthology.orchestra.run.smithi188.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 20 02:55 /home/ubuntu/cephtest 2023-07-20T02:55:01.219 INFO:teuthology.orchestra.run.smithi007.stdout: 524293 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 20 02:55 /home/ubuntu/cephtest 2023-07-20T02:55:01.223 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-07-20T02:55:01.233 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-07-20T02:55:01.253 INFO:teuthology.nuke:Checking targets against current locks 2023-07-20T02:55:01.284 DEBUG:teuthology.nuke:shortname: smithi007 2023-07-20T02:55:01.284 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-20T02:55:01.315 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344290', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 02:05:29.074186', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-20T02:55:01.316 DEBUG:teuthology.nuke:shortname: smithi158 2023-07-20T02:55:01.317 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-20T02:55:01.339 DEBUG:teuthology.nuke:shortname: smithi188 2023-07-20T02:55:01.340 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-20T02:55:01.346 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi158.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344290', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 02:05:29.075989', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-20T02:55:01.364 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi188.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344290', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 02:05:29.071809', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-20T02:55:01.382 INFO:teuthology.orchestra.console:Power off smithi007 2023-07-20T02:55:01.384 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-20T02:55:01.413 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T02:55:01.431 INFO:teuthology.orchestra.console:Power off smithi158 2023-07-20T02:55:01.432 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-20T02:55:01.453 INFO:teuthology.orchestra.console:Power off smithi188 2023-07-20T02:55:01.454 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-20T02:55:01.465 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T02:55:01.479 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T02:55:05.466 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T02:55:05.492 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T02:55:05.507 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T02:55:09.582 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T02:55:09.607 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T02:55:09.618 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T02:55:09.795 INFO:teuthology.orchestra.console:Power off for smithi007 completed 2023-07-20T02:55:09.821 INFO:teuthology.orchestra.console:Power off for smithi158 completed 2023-07-20T02:55:09.834 INFO:teuthology.orchestra.console:Power off for smithi188 completed 2023-07-20T02:55:09.940 INFO:teuthology.run:Summary data: client.0-kernel-sha1: a440c79eee0258ae07e4988d8690fe6187b8b7a3 description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsync-tester}} duration: 1899.205129146576 failure_reason: 'Command failed on smithi007 with status 2: ''sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name=qa sv_0''' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=6390ba8843e247af9294e3a42eb98e38 status: fail success: false 2023-07-20T02:55:09.940 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-20T02:55:10.017 INFO:teuthology.run:FAIL